Running a database is hard. Running a database in Kubernetes is harder. So why do it?
For me, the answer comes down to my experience with Vitess, which is an open-source, distributed clustering system used by companies like Slack to scale MySQL endlessly in the cloud. While working on Vitess at YouTube, I saw first-hand how running on Google’s Borg platform made large distributed systems, like our Vitess cluster with many thousands of MySQL nodes, feel magically simple for a small team to develop and operate. Unfortunately, our attempts to help users outside Google adopt the open-source Vitess system suffered from the lack of a similarly-capable, industry-standard abstraction layer for building distributed systems.
That all changed in 2014 when Kubernetes was announced; I was so excited that I immediately started working on porting Vitess from Borg to Kubernetes. However, I quickly found that there was still a lot missing. Kubernetes was on version 0.4 at the time and it essentially lacked any support for stateful workloads; there were no Persistent Volumes and no controllers to monitor and replace Pods. I continued polishing the port as Kubernetes slowly improved, but things got really interesting when CoreOS introduced the concept of Kubernetes Operators in 2016.
By that time, I had transferred to the Kubernetes team at Google to get my hands dirty helping to shore up support for running databases. Having learned so much from the authors of core Kubernetes components like Deployment and StatefulSet, I loved the idea of using CRDs (called TPRs at the time) to let people build higher-level, declarative APIs the same way we built the primitives in core Kubernetes. I developed Metacontroller in 2017 as a way to explore these patterns and make them more approachable.
The Year of the Database in Kubernetes
Finally, in 2019, I felt like the necessary pieces were in place to recreate that magic I had experienced running Vitess on Borg: a relational database that felt like an ever-expanding, inexhaustible resource, yet wieldable by a single person. I joined PlanetScale and started working on a Kubernetes Operator for Vitess, a project I had spent the last 5 years training to build.
It’s been quite a journey, but today I’m happy to announce, at long last, the release of our Vitess Operator. Please give it a try and let us know what you think. My hope is that you'll agree that this was a hard problem worth solving.
We open-sourced (Apache 2) the Vitess Operator that allows you to easily deploy Vitess in your own Kubernetes clusters.
Deploying databases in Kubernetes has many benefits including:
- Scale: Deploy Vitess clusters to scale databases horizontally or vertically
- High Availability: Replicate data across multiple Availability Zones to support immediate failover to recover from loss of an Availability Zone.
- Backups: Manually trigger backups with standard Vitess tooling.
To get started with the Vitess Operator, you can try out these getting started examples on the Vitess website and review the documentation here. The operator is also available from Digital Ocean as a part of the Digital Ocean Marketplace.
It’s worth noting that PlanetScale also makes available a battle-tested closed source version of the operator that supports more monitoring, security, customized scheduling, and automation features. In fact, we use it to run our PlanetScaleDB service across three clouds and 12 regions. To learn more about the PlanetScaleDB Operator, contact us.