A Tale Of Two – Well, Three Or Four – Kubernetes

It is safe to say that VMware would have been perfectly happy if Docker containers had never been invented. A decade ago, during the Great Recession, the company’s server virtualization platform matured just enough to be useful for server consolidation on a tectonic scale, allowing enterprises to forego a server upgrade cycle – or two – at precisely the moment when the CIOs of enterprises the world over slammed on the brakes for server spending.

With the ESXi hypervisor, the vSphere tools to activate its inherent capabilities, and the vCenter management console, server virtualization for Windows Server and Linux became practical and, while expensive, made economic sense over the short and long haul as machines that were running at maybe 5 percent or perhaps 10 percent utilization were virtualized and then run side by side on servers that were getting 60 percent or higher utilization rates. The fact that this VMware stack was pretty good at virtualization meant that VMware quickly became the front runner in virtualizing the vast base of Windows Server and Linux workloads in the enterprise. Hyperscalers either ran bare metal or had their own homegrown container environments, and the cloud builders all had their own server virtualization stacks. They didn’t need VMware and they didn’t use it because the ESXi stack is not open source and it is not cheap to license or support. Enterprises, by contrast can’t generally afford to develop their own systems software and they don’t mind paying a premium for the stuff so long as it works. They are more concerned with mitigating risks than they are lowering costs to the absolute lowest level and increasing control to the highest level.

The advent of containers by Docker, based on substantial work that was done by Google as cgroups and namespaces in Linux for providing something less heavy than a virtual machine for isolating workloads, really messed up the whole server virtualization story. VMware had long since hoped that the virtual machine would become the shrink-wrap container for software distribution, and here was this light-weight container usurping its hegemony. There was a lot of talk about running containers atop virtual machine hypervisors for security, which is how we ended up with vSphere Integrated Containers, one which doesn’t run Docker containers within an ESXi VM so much as run a Docker container as an ESXi VM, with the Docker engine embedded in the hypervisor.

VMware had a container stack called the Photon Platform, which was a greenfield Docker container system and orchestrator that was intended to take on Mesos, Kubernetes, and Docker Swarm. The Photon Platform included the lightweight Linux kernel called Photon OS, a stripped down hypervisor based on ESXi that is called the Photon Machine as well as a controller akin to Kubernetes (but not based on it) called Photon Controller, both of which were open sourced a few years back. Eventually, Kubernetes as a service was added to the Photon Platform in November 2016, swapping out the homegrown controller for the Kubernetes one open sourced by Google a few years back and very quickly becoming the container orchestration standard. This whole stack was meant to run atop NSX virtual networking and vSAN virtual storage.

The Pivotal division of Dell that originally created its own Hadoop data analytics distribution and where the Cloud Foundry platform cloud software ended up now has its own Pivotal Container Service, shortened to PKS to let you know it means Docker containers orchestrated by Kubernetes. This software also rides atop vSAN and NSX, and it is intended to be the version of a container service from the partnership of VMware and Pivotal (now a public company still majority owned by Dell, just like VMware) that is meant to be installed on-premises in corporate datacenters. But it can also be run on Google Cloud Platform, thanks to a partnership between Google and Pivotal.

If you can keep track of this, you are doing better than most.

But wait. It is actually going to get easier even as it gets more complex. This week, VMware announced the VMware Kubernetes Engine, or VKE, for short, and it provides a container service that is a clean slate approach to developing a Kubernetes control freak that is distinct from Photon Platform or Pivotal Container Service. This VMware Kubernetes Service is not available on the VMware cloud that runs the entire SDDC stack – vSphere, vSAN, NSX, vRealize – on Amazon Web Services hardware as far as we know.

But what it is intended to do, according to Bill Shelton, vice president of product management of cloud native applications at VMware, is provide a Kubernetes service running on the AWS public cloud today and the Microsoft Azure public cloud sometime in the near future.

Here’s the block diagram describing all of these things, with Photon Platform no longer in the mix:

The neat bit about VMware Kubernetes Engine is that it includes a feature called Smart Cluster, which takes input from operators about a workload and the required service levels and then automagically sets up the services on which Kubernetes and Docker will run. The other neat bit is that it runs natively on the AWS cloud, and not on top of the VMware stack. It will run natively on Azure at some point.

We know what you are thinking. This raises a lot of questions. Why does Google Cloud Platform have to run Pivotal Container Service, which requires people to build and operate it, and only AWS and Azure get this new VMware Kubernetes Engine that is operated by VMware? Moreover, why isn’t VMware Kubernetes Engine based on Pivotal Container Service, just a variant that VMware itself supports? How come Google Cloud Platform is not getting VMware Kubernetes Engine , and moreover, how come VMware Kubernetes Engine is not being offered on-premises on clusters running the VMware stack inside enterprise datacenters? And most importantly, why hasn’t VMware created a single substrate that can span public clouds and private clouds and provide a single, consistent, easy way of using Kubernetes, which can be run by enterprises if they choose or by VMware if they choose?

No offense, and we think it is great that VMware is creating an easier way to use the Kubernetes orchestrator and Docker containers, but it has forked and overlapped its product lines so much now in the container realm that it is beginning to look like the IBM systems of the 1980s and 1990s, when Big Blue could not decide what it was and what it wasn’t.

[Editor’s note: Having read this piece, our colleague Dan Robinson piped up with this input: “One thing I found out during my scratching around is that PKS is built on KUBO, Pivotal’s version of the BOSH deployment and lifecycle tool, which it has adapted for Kubernetes. The thing is, BOSH has an abstraction layer called the Cloud Provider Interface (CPI), through which it manages resources in the infrastructure layer. This is how PKS is able to run on both vSphere and Google Cloud – because there are CPI configurations for both. The interesting part is that the CPI already supports AWS and Azure, meaning that VMware/Pivotal could have released a version of PKS to run on these clouds as well, but chose not to. Why? The mystery deepens.”]

The idea, if there is one, is to be one container service to rule them all. It is amazing to us that VMware is not trying to produce that kind of substrate, abstracting away the complexity much as it did with server virtualization. Maybe VMware and Pivotal should have merged and figured this all out. In fact, we thought that had already happened. Twice now that Dell bought the whole shebang. The good news is that this gives VMware something to sell to companies that want to create a Kubernetes substrate that that spans AWS and Azure. Perhaps Google Cloud Platform at some point. The bad news is that Google Cloud Platform is not supported by VMware Kubernetes Engine , and neither is on-premises iron, from which VMware gets the vast majority of its revenues from its 500,000 customers worldwide.

In any event, VMware Kubernetes Engine has been in beta for three months, and will be generally available sometime in the second half of this year. Pricing is not yet available, but without that whole SDDC stack in there, it should not be very expensive.

In addition to rolling out VMware Kubernetes Engine, the Pivotal Container Service was upgraded to support Kubernetes 1.10, and also added support to span multiple availability zones in Google Cloud Platform to increase high availability. vRealize Log Insight and Wavefront (which was acquired and later integrated with the Photon Platform) have been integrated with Pivotal Container Service to increase the monitoring of Kubernetes itself and the containers under its care and feeding.

Sign up to our Newsletter

Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.
Subscribe now

1 Comment

  1. And another… VMware also ships Kubernetes with VMware Integrated OpenStack (VIO) since version 4.1. It includes a few different deployment configurations.

Leave a Reply

Your email address will not be published.


This site uses Akismet to reduce spam. Learn how your comment data is processed.