Mesos Borgs Google’s Kubernetes Right Back

The rivalry between Mesos, Kubernetes, and OpenStack just keeps getting more interesting, and instead of a winner take all situation, it has become more of a take what you need approach. That said, it is looking like Kubernetes is emerging as the de facto standard for container control, even though Google not the first out of the gate in open sourcing Kubernetes and Docker Swam and the full Docker Enterprise are seeing plenty of momentum in the enterprise.

Choice is a good thing for the IT industry, and the good news is that because of architectural choices made by those who are creating virtualized infrastructure, choice is possible. The whole point of containers is that the provisioning of runtimes for both stateless and stateful applications and the provisioning of the underlying compute, storage, and networking – whether they are virtual or physical – that supports those containers are abstracted away from each other and can be managed and monitored separately.

Kubernetes is, of course, an open source container management system that was inspired by the Borg cluster controller and container management system that the search engine giant created more than a decade ago. Early versions of Borg provisioned bare metal servers in its clusters and it was not until 2005 that Google started hacking together Linux software containers to try to drive up utilization on its vast infrastructure. A lot of the work that Google did to make Linux work better containers – mainly cgroups and namespaces – was initially employed by the founders of the Docker container format and its runtime, but Docker has since moved to its own code. (Linux containers are still a part of all commercial Linux releases.) This container work was also derived from Google’s proprietary platform, and the combination of LXC containers or Docker containers plus Kubernetes and an underlying infrastructure provisioning system (be it OpenStack plus Magnum or Pivotal BOSH plus VMware vSphere or CoreOS Linux and Tectonic, just to name three) is the closest thing to the Google Borg stack that an outsider can have.

Mesos comes at hyperscale infrastructure from a slightly different angle. If Google is tightly linked to Stanford University, then Mesos is coupled to the University of California at Berkeley’s AMPLab, which is also where the Spark in-memory processing framework hails from, and its initial engineering work was done by software engineers at Airbnb and Twitter, who went on to found Mesosphere in an effort to expand it from a sophisticated provisioning system for bare metal clusters to a true platform that can management multiple software stacks running side by side on clusters. The commercial version of Mesos is called the Data Center Operating System, or DCOS, and it comes in a basic open source version now plus an enterprise edition with some extra goodies that companies have to pay for. Last July, the Apache Mesos project, which has the core Mesos controller, hit its 1.0 release milestone, meaning it was ready for production installations. At that time, Mesos added support for native Docker runtimes on top of the cluster controller. Prior to that, Mesos embedded its own container controller system, based on its own variant of the cgroups and namespaces technology open sourced into the Linux kernel by Google, into the Marathon application framework.

Not everyone could wait for Mesosphere to get a native Docker orchestration system, or better still support for Kubernetes as it took off in the market. While most of Mesosphere’s more than 100 commercial customers use Marathon as their container controller, the “Wave 0” early adopters of the open source Apache Mesos code created their own container orchestrators to ride atop Mesos. Twitter built Aurora, which is itself now an Apache project. Apple built Jarvis, the container framework that underpins the Siri personal assistant application for iPhones. Netflix built Titus, which runs atop Amazon Web Services because all Netflix infrastructure does. HubSpot built singularity. Bloomberg has hacked together its own controller while using native Marathon containers, as we have previously detailed. And Verizon is doing the same thing, at least for now.

Tobi Knaup, co-founder of Mesosphere and its chief technology officer, created the search engine and fraud detection software at house sharing juggernaut Airbnb and also created the Marathon application framework that is a key part of Mesos, and we reminded him that way back in the day when Mesos was just starting to be commercialized, Kubernetes was actually a supported container controller for Mesos and Mesosphere engineers were among the top contributors to the Kubernetes effort, so announcing support for Kubernetes, as Mesosphere is doing this week ahead of the MesosCon event next week, is not precisely new.

As it turns out, that initial integration was less than ideal, and it was deprecated for a time. Mesos itself is not a container orchestration system, and most customers who wanted to run containers, whether the native Linux containers that Mesosphere created or the Docker layer added to Marathon (and the first orchestrator for Docker containers, ahead of Docker Swarm and Kubernetes), did so without Kubernetes.

“Mesos was created as a low level resource management program that manages any kind of workload, including container orchestration, to run on top of it,” Knaup explains to The Next Platform. “It has always been our view that customers should have a choice about what tools they want to use on the platform. We built Marathon ourselves, but we want to give customers choice, and that is why we decided early on to invest in Kubernetes. Back then, Kubernetes was a young product, it was moving fast, and it did not have stable APIs at the time, so keeping that integration stable and getting it to something that we could recommend to a customer to run in production was hard. We are a startup, and this was just not sustainable. So we mothballed that effort and focused on our platform and specifically on a software development kit that would bring Kubernetes and a lot of other software – such as Kafka, Cassandra, Spark, and HDFS – to that platform more easily and cleanly. So we were able to restart this Kubernetes integration earlier this year using this new SDK, and it is a much cleaner integration and also allows us to take the official, upstream Kubernetes releases and bring them to DCOS very quickly. We are not performing any brain surgery on Kubernetes or adding any proprietary bits, and this is important.”

To be precise, DCOS 1.10 will have this new Kubernetes support added to it in beta form, and it will use Kubernetes 1.7.3. As Google puts out new Kubernetes, Mesosphere will intersect them, and the expectation is that when DCOS 1.11 ships later this year, it will have production grade support for the then-current Kubernetes.

The native Kubernetes support inside of Mesos will probably help accelerate adoption of DCOS, in either its open source or commercial form. Knaub estimates that there are many thousands of customers that have deployed Mesos, either the core Apache Mesos variant or the open source DCOS variant from Mesosphere, in production; there is no way to be sure how many proofs of concept are running out there, but the number is probably multiple integer factors larger than the installed base, if not an order of magnitude if the history of other popular open source projects related to infrastructure software is any guide. (Think Eucalyptus in the early days, or OpenStack after that, for controllers for virtualized private clouds.)

Mesosphere is approaching 300 employees, and has raised a total of $112.3 million in four rounds, the last of which included big sums from Hewlett Packard Enterprise and Microsoft. Mesosphere has made no bones about the fact that it is targeting the Global 2000 first with DCOS, but obviously such advanced technologies eventually make their way into smaller enterprises and other institutions, such as HPC centers and service providers. Five of the ten largest financial services firms in the United States use Mesos, and five out of the ten largest banks in North America are doing so, too, and so are five of the ten largest telcos worldwide. Six of the biggest automakers who are building platforms to support connected cars have also chosen Mesos (in one form or another) as their core platform. The mix of Kafka for data ingestion and stream processing, the HDFS distributed file system for long term storage of data, the Cassandra NoSQL database for fast access to structured data, and the Spark in-memory processing seems to be the preferred stack.

By the way, there is no official integration between the OpenStack cloud controller, which provisions hypervisors on bare metal servers and virtual machines on top of that, and Mesos, which is a broader kind of cluster management and provisioning system. But a number of companies that standardized on OpenStack years ago are running Mesos atop it, just like you can run Mesos atop AWS cloudy infrastructure. It is funny to contemplate what might happen if customers ran OpenStack on top of Kubernetes on top of Mesos – and it is technically possible. Conversely, you could run Mesos on top of OpenStack with Kubernetes on top of that.