Everywhere we turn in the datacenter, monolithic software is being smashed into smaller chunks that can be isolated from each other and tweaked as needed without having to recompile and test a giant pile of code. This approach to writing software is new to a lot of enterprises, but for the hyperscale and web application providers that grew up in the second wave of distributed computing, stack up applications that are comprised of hundreds to thousands of microservices is just the way things are done.
VMware, the juggernaut of the server virtualization wave that swept over the X86 server portion of the datacenter in the past ten years and that has made the company one of the richest and largest software companies in the world, cannot just sit idly by and let this new form of application virtualization pass it by. And in fact, the company is taking some lessons from Google and other hyperscalers and creating a variant of its virtualization stack that plays well with containers and microservices. And, perhaps somewhat surprisingly to some, the company has cooked up its own minimalist Linux operating system, called Project Photon, that it will be giving away for free with its ESXi hypervisor so companies can use it as the foundation for many different styles of virtualized software containers.
The interesting thing about these new-fangled applications is that they are not written with the assumption that the underlying infrastructure has to provide rock-solid availability. The assumption is that you have multiple instances of each microservice and if one instance of that service or a whole server in a cluster crashes, the application has the smarts to recover from that failure and keep the overall application running. The software container is just a runtime for these microservices. That is not to say that companies don’t want to also do all they can to keep the underlying infrastructure running – in this case, multiple instances of a Linux operating system that run the containers – and hence companies will mix and match bare metal server virtualization and software containers in different ways to get the availability, resource isolation, and security levels they need. This is, in fact, how Google runs its Compute Engine infrastructure cloud. Google lays Linux software containers based on namespaces and cgroups as a base foundation, then puts a KVM hypervisor on top of that, and then another Linux container actually provides the server slice on the public cloud.
To get a handle on this new way of building software, VMware has created a new team focused on cloud-native applications, and the new Project Photon minimalist Linux and Project Lightwave identity and access management software that the company is open sourcing is the first of a line of tools that the company has put out from this team. VMware has also hooked up with sister EMC company Pivotal to integrate these tools with Project Lattice, a lightweight variant of the Cloud Foundry platform cloud that is initially be targeted at developers who want to be able to quickly stand up platform clouds using software containers and hypervisors as they write their distributed, virtualized, and containerized applications. While Lattice is not intended as a production environment, that does not mean for a second that it will not be used as such. In fact, you can almost bet that developers will try to put Lattice into production just to see what it can do compared to the full-on Cloud Foundry.
You might be wondering why the open source minimalist Linux and identity management software for containerized applications is coming out of VMware, which doesn’t generally do open source software, rather than from Pivotal, which has a lot of open source software and which is in the process of opening up most of its core software, in fact. (Just last week, Pivotal open sourced its GemFire in-memory database as Project Geode.) Mike Adams, director of product marketing at VMware, tells The Next Platform that VMware tends to concentrate on the infrastructure layer and Cloud Foundry and the other Pivotal tools tend to be higher up the stack, abstracting this infrastructure as a platform cloud. Hence, the minimalist Project Photon Linux, however paradoxical it might seem, belongs at VMware, not Pivotal. Ditto for the identity and access management software that VMware has cooked up for containers.
As Light As A Photon
Photon is a lightweight Linux operating system that VMware built from the ground up to specifically support various container formats. The idea is, which CoreOS and Red Hat have done with their own Linux distributions, is to cut out all of the unnecessary bits of the operating system that are not needed to run containers, which makes the Linux easier to maintain, run in a smaller memory and disk footprint, and easier to secure.
Photon was created starting with the Linux 3.19 kernel for X86 machines, and Jared Rosoff, senior director of engineering at VMware, tells The Next Platform that Photon Linux was created specifically to run atop the ESXi hypervisor as a guest operating system and that it is not intended to run on bare metal machines. It is not derived from any other Linux distribution, but is rather unique to VMware.
Just to give you an idea of how skinny Photon is, its disk image is about 300 MB, which is an order of magnitude smaller than the 3 GB disk footprint of a typical Linux server distribution. “The memory footprint for Photon is smaller in the sense that it is not running a lot of services that would normally boot up,” Rosoff says. “They have various servers and daemons running, and Photon is just focused on the core set of services to get a container runtime up and running.”
Photon is going to be open sourced under a GPL v2 license. The plan is to deliver Photon free of charge and bundled with the ESXi hypervisor, but it is entirely possible that Photon and its companion Lightwave could end up being mixed with ESXi and Lattice from Pivotal to create a new containerized stack that is distinct from vSphere and therefore with its own support contracts and pricing for those who want hand holding from VMware. VMware is going to be making other announcements this year relating to cloud native application development and deployment, says Adams, and it could clarify at that time. “We could have other components that we want to deliver as part of this cloud native application infrastructure,” he hints.
It would be interesting to see ESXi itself opened up as part of the stack, but there is no indication that VMware would open source the family jewels. It could, however, open source a subset of the ESXi hypervisor much as it has done with NSX to create OVN. Then it could offer a completely open source container stack – something it may need to do to get penetration is market segments that increasingly have open source as a requirement. This is why, after all, sister company Pivotal is opening up its entire platform and database stack.
The other important thing is that VMware will be doing all of the certification for common third party software applications to make sure that they are certified to run on Photon, much as it does to certify applications using Windows or Linux run adopt ESXi already.
“We think it actually over time Lattice can become a lightweight competitor to the full Cloud Foundry platform. We are excited to see how that goes, and we would rather disrupt ourselves with a single tenant version than not give that to users.”
Project Photon Linux offers RPM-based imaged-based versioning as well as Yum-compatible package-based management, which is useful. It supports Docker container and CoreOS App Container/Rocket (now called rkt) containers riding atop of it as well as Pivotal’s own Garden container format. The latter was created by Pivotal in the Go programming language as a means of managing Linux container (LXC) instances. CoreOS has signed up to distribute Photon Linux with its rkt containers, and VMware has joined the App Container community. Mesosphere is working with VMware to integrate Photon and Lightwave with its Mesos “data center operating system,” and JFrog and HashiCorp are packaging up Photon with their respective Atlas and Bintray services.
Pivotal is of course adding both Photon and Lightwave to its Lattice minimalist platform cloud, which has been in a private beta for about a month now. Lattice includes the “Diego” container scheduler from Cloud Foundry as well as the load balancing and log aggregation components of the software. It does not include high-end data services such as clustered MySQL and multitenant management.
James Watters, general manager of the cloud platform group at Pivotal, says that the normal Cloud Foundry platform cloud software has a lot of stuff in it for managing complex software stacks, and eats around 22 cores and 40 GB of memory on the Amazon Web services public cloud or on a private vSphere virtual cluster. It can scale to many tens of thousands of nodes. Lattice, by contrast, has only the essential bits of Cloud Foundry needed to run containerized applications and only a 256 MB memory footprint and can scale to thousands of nodes.
“We think it actually over time Lattice can become a lightweight competitor to the full Cloud Foundry platform,” says Watters. “We are excited to see how that goes, and we would rather disrupt ourselves with a single tenant version than not give that to users.”
Watters says further that application certification angle with Photon Linux will come into play for Pivotal. Watters says that historically, Cloud Foundry favored Canonical’s Ubuntu Server Linux, but the problem is that many third party systems and application software packages are not certified to run atop Ubuntu Server. Photon will be embedded to run on ESXi and basically any Linux application that is certified to run on ESXi will eventually be certified to run on Photon.
Streaming Photons Into A Lightwave
While the microservices style of programming has many benefits in terms of being parallel and resilient, the resulting software does have many more points of attack and security is therefore a bit trickier, says Adams. “If you are going to deploy a cloud-native application, you are going to need identity infrastructure as you scale to thousands of nodes and hundreds of application components. You need to have network isolation that allows you to be able to grow and scale very quickly. And you need to have a trusted compute runtime – are things that are running doing what they say that are doing and are they located where they are supposed to be, and do they have the trust level you require.”
The Project Lightwave identity and access management tool will be available to hook generic Linux setups that use the Open Virtual Network (OVN) software that VMware open sourced back in January. With OVN, VMware is taking a chunk of the functionality that is embedded in the NSX network virtualization software that it got a few years back when it acquired Nicira and opening it up to add functionality to the Open vSwitch that is part of the NSX stack as well. OVN adds virtual network abstractions like virtual L2 and L3 overlays and security groups to Open vSwitch, which as the name suggested in an open source virtual switch that plugs into a hypervisor and provides virtual switching for virtual machines running on the hypervisor. The OVN stack will also include gateways between virtualized devices and physical ones, since many enterprises have a mix of the two, and there is talk that it will eventually support multicast replications, which is done by hypervisors now.
Lightwave will hook into the open source OVN stack and interface with popular Linux distributions that are used as a substrate for containerized applications, as shown at the left in the chart above. And on the right of that chart, you see the variant of the containerized stack that VMware will be pitching to its enterprise customers, who may prefer to run containers atop a vSphere/ESXi server virtualization layer and who will want to deploy, so VMware hopes, the full-on NSX network virtualization. But, then again, customers can weave when vendors think they might bob. It could turn out that in the long run, VMware will offer a commercially supported version of the open source stack, too, because companies prefer this for whatever reason. (If you are curious, the OVN software is written in C, and Lightwave is written in a mix of C and Java, according to Rosoff.)
The Lightwave software provides single sign-on, authentication, and authorization of containers and the orchestration tools that manage the containers using user names and passwords, tokens, and certificates, and adopts a number of popular access standards including Kerberos, LDAP v3, SAML, OAuth 2.0, OpenID Connect, X.509, and WS-Trust. The Lightwave server is implemented as a clustered master node, so it can scale out without falling over, and it can also scale across multiple datacenters. Lightwave has multitenancy features so multiple business groups within a company or multiple companies on a cloud can be hosted on the same infrastructure and still be isolated from each other. Lightwave is being opened up under an Apache 2.0 license, which means it is compatible with a wide variety of open source software; it will be available before the end of the second quarter of this year. OVM is opened up under a GPL v2 license. Lightwave incorporates some of the functionality of OpenLDAP, but Rosoff says that a lot of the code is culled from the Platform Services Controller inside vSphere 6.0 and is “hardened and battle-tested.”
So maybe the idea of open sourcing a lightweight ESXi does not sound so crazy after all.
Hi Timothy.
I like that VMware is working on a Linux option, not that we need another one, but I think there is more behind the curtain than a lightweight OS for containers. How about just the idea that VMware will soon have their own Linux Distro to court all the Linux admins who prefer building Centos and Ubuntu VMs on OpenStack?
Anyways, good article!