Technologies that are developed at hyperscale companies that operate applications at the datacenter scale, rather than of the server or the rack, can be tweaked to work for smaller enterprise customers. Software containers, which provide isolation between software stacks running on shared clusters that is much less heavy than full-on server virtualization with a hypervisor, is one such technology. The trouble is, commercial-grade container management systems are still not quite polished enough for enterprise use. But they are getting closer.
CoreOS, which has created a variant of Linux that can only be deployed in containers and that has been working with Google to create a container management system, is putting a beta of its first full, complete container management stack, called Tectonic, into public beta. Alex Polvi, co-founder and CEO of CoreOS, tells The Next Platform that he expects for Tectonic to ship sometime in the second half of this year, depending on the feedback from customers. Tectonic is based on a mix of tools that were developed in house at CoreOS and that come from search engine giant Google, which runs all of its software in containers. We are talking billions of containers on millions of servers, just to get a sense of the Google scale.
Google Inspired Software Shipping Containers
Google has been developing and perfecting container technology for more than a decade, starting with a crude form of workload isolation for Linux applications back in 2004 and then moving on to Linux control groups, or cgroups, in 2006. This cgroups code is the basis for Linux containers, or LXC as it is called inside of the major Linux distributions, which Google has donated to the open source community. In 2013, Google cooked up a variant of Linux containers, called lmctfy, short for “let me contain that for you” that it is pushing. And Docker has come out with another style of software container that is similar to but distinct from LXC and lmctfy and that is catching on like wildfire in the enterprise.
Just having containers is not enough, of course. If you create whole applications or microservices components of them and isolate them into hundreds, thousands, or more containers, you need something that can aggregate containers and manage them as a whole. Google has been using its homegrown Borg and now Omega cluster controller and job schedulers for its own internal applications as well as for its Cloud Platform public cloud. The company did not want to open source either Borg or Omega, since those tools are specific to its own workloads and they are also a key technology that gives Google its competitive edge. But at the same time Google believes that containers need to be a salient feature of its Cloud Platform, and that is why it created and open sourced a container management system called Kubernetes last year. The hope was that Kubernetes would become popular as a management tool for all kinds of containers and that by making sure that these tools worked well on its Compute Engine infrastructure cloud, Google would have an advantage over its rivals Amazon Web Services and Microsoft Azure.
One of the key partners that Google has as it tries to encourage adoption of containers and its Kubernetes management system is CoreOS, a maker of a minimalist Linux operating system aimed initially at hyperscalers that was founded in May 2013. (The Next Platform did a detailed analysis of CoreOS back in February.) CoreOS embraced Docker containers from the get-go, and in fact it was only possible to deploy the minimalist CoreOS Linux inside of Docker containers, and the reason for that is that CoreOS is a continually updated Linux platform and requires containers to make updates without disrupting application software. The founders of the CoreOS project are unhappy with the complexity and security model of Docker as it has evolved, and last year announced a new container format, called App Container, and a related management tool, called Rocket, that will work with App Container. The company is still supporting the Docker container format, and Polvi says there is no plans to change that. Both Docker and Rocket formats (it is now abbreviated rkt) are supported in Tectonic, despite reports to the contrary that only rkt formats were supported.
“You can think of different Tectonic regions conceptually as different regions on Amazon Web Services, and just point the tools at different regions and the applications deploy in a very consistent way. This is a dream that we have all been talking about for a while.”
The Kubernetes project has seen the same kind of enthusiasm among cloudy infrastructure suppliers that are trying to catch the container wave at the same time as maintaining their server virtualization stacks. Kubernetes has been woven into Red Hat’s Atomic Host container management system, which was launched in February. VMware has tapped it to be the Docker container management layer that interfaces with its vCenter Server management console for its vSphere/ESXi server management stack. Mesosphere has commercialized an analog to Google’s Borg job scheduler called Mesos, created by techies at Airbnb and Twitter, and is layering Kubernetes on top of it as a container management system. Microsoft is adding Docker container support to Windows and has joined the Kubernetes effort as well and will presumably be weaving it into its System Center tool; for the moment, it is making sure that CoreOS runs well with Kubernetes on its Azure public cloud with Linux instances. IBM has joined the Kubernetes effort as well and has partnered with Docker to make sure its containers are “enterprise grade,” and has Docker orchestration services based on Docker Hub in its BlueMix variant of the Cloud Foundry platform cloud. It is unclear if IBM will simply do a more streamlined Kubernetes stack on top of the Linuxes for its Power Systems scale-out machines, but it could do that fairly simply by offering the Tectonic stack on top of its Power8 iron.
Kubernetes and Docker will no doubt get some play in the HPC arena because of the relatively low overhead of containers versus hypervisor-based server virtualization, as we discussed recently.
Almost Ready For Prime Time
The Tectonic stack includes the Kubernetes container management system as well as the complete stack of tools developed by CoreOS to manage clusters and the Docker containers that run on them.
“This gives customers an end-to-end container infrastructure,” Polvi tells The Next Platform. “We are really using everything we’ve got – CoreOS, etcd, flannel, fleet – and then putting a Kubernetes control plane on top of that. Companies that are taking our components are building something like Tectonic by piecing it all together.”
For those of you not familiar with the CoreOS products, etcd is a distributed Linux configuration information tool that works with CoreOS and puts all of the configuration information for the many nodes in a cluster into in a key-value store that can be rapidly searched. (Google picked the etcd software as an underpinning of Kubernetes configuration and Mesosphere uses it to house configuration data for Mesos as well.) The fleet service for CoreOS places containers and their applications atop the CoreOS clusters, and flannel is a network fabric for containers that runs atop etcd. In addition to these open source pieces, CoreOS has added an installer, an updater, a web-based management interface, and a single sign-on interface to forge Tectonic. Kubernetes allows for collections of software containers to be grouped together and managed as a whole, in essence creating a virtualized and containerized application.
Polvi compares the Tectonic stack to the Android Linux environment that Google created for smartphones and tablets. The idea is to create applications that will then install on top of this CoreOS, Docker, and Kubernetes stack, and one of those applications created by CoreOS is an on-premise container registry, appropriately called Enterprise Registry. (CoreOS acquired this registry service last year when it bought a company called Quay.)
Pricing for the Tectonic containerized application stack has not been announced yet and will be finalized when it becomes generally available. If you are interested in testing out the software, you can sign up at this link. CoreOS has a handful of beta testers at the moment, including a cloud service provider, a financial services company, and an ISV that is running its applications in SaaS mode. Eventually, the beta program will be opened up a bit wider, but for now, CoreOS is being selective on who gets to test it out because it is still a relatively small company and because it wants to get a diversity of test cases.
At the moment, CoreOS sees most of the installations of its Linux variant on the AWS public cloud, with on-premises gear being the second most popular installation option, and this is where CoreOS is focusing the testing of the Tectonic stack at the moment, although you can run on Microsoft Azure or Google Cloud Platform. The latter is important because, as we pointed out above, Google wants to foster the use of containers on its Compute Engine infrastructure cloud, and it specifically has created and helped commercialize Kubernetes to this end.
While Tectonic can run on-premises or in the public cloud and it can treat containerized software infrastructure as a single management domain and run it all underneath one Tectonic instance, you cannot live migrate running containers back and forth between cloud and on-premise machinery, or between clouds. But Polvi says that this capability would be “a very natural thing for us to do,” and says that in the meantime, the application portability that containers offer is a big step in this direction.
Having a unified container format that isolates application components from the underlying operating system that those applications run on is why LXC and Docker containers were invented in the first place, and it is why zone containers for Solaris and OpenVZ/Virtuozzo containers for Linux were created even further back in the past. Docker will eventually span both Windows and Linux, making it unique among container formats.
“Because we are managing at the container level, the application level, application portability becomes a true option,” says Polvi. “You can think of different Tectonic regions conceptually as different regions on Amazon Web Services, and just point the tools at different regions and the applications deploy in a very consistent way. This is a dream that we have all been talking about for a while.”
The irony is that containers may not replace virtual machines, when all is said and done. For one thing, public clouds are built using hypervisors and that is not going to change any time soon; containers get layered on top of virtual machines. Or, in the case of Google Compute Engine, homegrown Linux containers are laid down on bare metal, then KVM hypervisors are loaded on top of these to provide security and isolation, and then tools like Tectonic will be used to create containers atop the KVM virtual machines.
Containers and virtual machines both have their uses, for the moment. But there could come a day when virtual machines as we know them go by the wayside – particularly as applications are composed of containerized microservices rather than big wonking hunks of monolithic C or Java code.
Google is not just standing behind CoreOS as a technology partner, but is literally helping fund the business. CoreOS has raised a total of $8 million in prior funding rounds, and with the funding announced today in conjunction with the beta release of the Tectonic platform, Google Ventures, the equity investing arm of the search engine giant, is leading a $12 million funding round. Kleiner Perkins, Fuel Capital, and Accel Partners, who invested last June in the Series A funding for CoreOS, all kicked in dough this time around, too. CoreOS got an undisclosed seed round from Y Combinator in March 2013 and another seed round which it did not reveal the size of in October 2013 from Andreesen Horowitz, Fuel Capital, and Sequoia Capital.
Polvi says that CoreOS is growing fast. It had 20 employees in December and as of this week it is up to 36. That cash will be used to get Tectonic ready for general availability and to hockey stick the company’s growth.