Containers Killed The Virtual Machine Star

We predict new enterprise application development will pass a tipping point in 2019 and shift away from legacy virtual machines (VMs) and strongly toward containers and Kubernetes container orchestration.

To be precise, we predict that:

• The future is multi-cloud, and multi-cloud means Docker containers with Kubernetes orchestration. Every public cloud has its own APIs, and in that sense, they are all new versions of proprietary mainframes.
• “Lift-and-shift” of VMware virtual machines will be more expensive than customers realize. Paying no money up front, in this case to refactor and port applications to Kubernetes, typically means paying more during operations.
• Java is not dead. It may play an important middle ground between lift-and-shift and the expense of completely refactoring applications for cloud-native environments. Java may be a light touch version of “move-and-improve.”
• It seems likely that Lenovo will take a look at acquiring SUSE, with Supermicro perhaps also in the mix.

Amazon Web Services, Cisco Systems, Google, IBM, Microsoft, Oracle, Red Hat, and VMware all recently announced strategic product plans. Their playing pieces are on the game board for 2019. The Cloud Native Foundation’s KubeCon/CloudNativeCon events featured two more private cloud announcements: Dell Technologies take-over of VMware’s stock symbol has been approved by shareholders and has taken effect and IBM made a bid for Red Hat that Red Hat’s shareholders will vote on in January.

What is a Private Cloud?

Private clouds are built on an infrastructure-as-a-service (IaaS) or platform-as-a-service (PaaS) base. A true private cloud must have these attributes:

• IT customer self-service. This is a central tenet of any cloud service provisioning.
• Secure single-tenant solution. Private cloud must be private to the organization paying for it.
• Transparently scale application resources. For hybrid cloud business models, the same application image must run in both private and public infrastructure. Bonus points if applications can scale from private cloud to multiple public clouds (multi-cloud).
• Data and application maintain proximity wherever they run. Software-defined networking (SDN) and software-defined storage (SDS) are key differentiators from legacy on prem IT architectures.

OpenStack And Red Hat OpenShift

In case you thought OpenStack had somehow disappeared for private cloud, it has definitely not gone away.

• Huawei’s FusionCloud Private Cloud Solution, Inspur’s InCloud and Tencent Cloud’s TStack Solution all are native implementations of OpenStack, as are Huawei and Tencent’s public cloud frameworks. Huawei FusionCloud Container Engine supports native Kubernetes in its public cloud.
• Supermicro certifies Canonical and SUSE’s OpenStack distributions.
• Lenovo certifies SUSE Containers-as-a-Service (CaaS) platform, which runs on OpenStack and public IaaS and PaaS clouds.
• Red Hat built its OpenShift product, a container solution implementing Docker and Kubernetes, on top of its OpenStack distribution. Red Hat’s OpenStack and OpenShift distributions are certified by most public and private clouds, as well as by most server manufacturers worldwide.

Red Hat recently introduced its OpenStack Bare Metal Certification. Server vendors may now certify that their products can be managed by OpenStack Bare Metal (the productized OpenStack Ironic project). OpenStack Bare Metal manages servers remotely. It is compatible with server management solutions from vendors such as Dell EMC ‘s iDRAC and Hewlett Packard Enterprise’s iLO, plus the Intelligent Platform Management Interface (IPMI) and Redfish. This is a great move that should accelerate enterprise adoption of Red Hat’s OpenStack and OpenShift products in private clouds.

However, in late October 2018 IBM announced its intent to acquire Red Hat (Red Hat shareholders will vote on the acquisition in January). The pending acquisition is spurring IBM’s datacenter infrastructure and public cloud competitors (i.e. everyone, much like Dell and VMware) to evaluate alternatives to Red Hat’s OpenStack distribution and to OpenShift.

Kata Containers and AWS Firecracker

Kata Containers are lightweight (low resource usage) QEMU-based VMs designed to run Docker and Kubernetes on the OpenStack framework. The OpenStack Foundation announced its Kata Containers project and community working group at last year’s KubeCon. Since then the Kata Containers project has made noteworthy progress, especially with China-based CSPs such as Huawei and Tencent, although no private cloud products have been announced yet.

Kata Containers running on OpenStack distributions was a great idea, until AWS released its Firecracker code as an open source project at its re:Invent conference in late November. AWS Firecracker is fast and secure micro-VM that has a lightweight resource use profile.

AWS built Firecracker on Linux’s kernel-based virtual machine (KVM) and is an alternative to the QEMU virtual machine manager. Firecracker can theoretically run on top of OpenStack, but AWS has only focused on container runtimes (specifically containerd and Kata Containers). Firecracker is also battle-hardened, as it is already deployed underneath AWS’s Lambda and Fargate production public cloud services.

Firecracker appears to be a direct competitor to Kata Containers, however, the Kata Containers project quickly responded by introducing early support for Kata Containers using Firecracker as an alternative to QEMU. While AWS isn’t likely to focus on OpenStack, supporting Kata Containers on Firecracker from the OpenStack side might enable movement of Kata Containers between OpenStack deployments on AWS. We’ll see how each project evolves over the next couple of quarters.

Public Cloud On Premises

This year, Google Cloud and AWS joined Alibaba Cloud, Baidu Cloud and Microsoft Azure in wedging a part of their bespoke public cloud software into a private cloud footprint capable of being deployed on prem within one server rack.

Alibaba Cloud launched its Container Service for Kubernetes this year. Alibaba designed its own PouchContainer container system but is in the process of migrating from PouchContainer to Kata Containers. However, it has not yet announced explicit Kubernetes support in its Apsara Stack Enterprise private cloud distribution. Apsara Stack Enterprise already supports Docker, so we expect Kubernetes support in 2019. Customers can install Apsara Stack Enterprise on existing servers.

Baidu Cloud supports Kubernetes and Baidu Cloud’s ABC-STACK private cloud framework supports IaaS, but it is unclear yet whether ABC-STACK supports containers or Kubernetes. Baidu Cloud ships ABC-STACK on specially configured ABC servers preinstalled in racks based on the Project Scorpio specification.

Microsoft Azure started supporting Red Hat’s Open Shift Container Platform in Azure Stack in September.

Microsoft recently made Kubernetes Marketplace available in preview status for Azure Stack. Microsoft Azure previews have reduced or different service terms than production software. Kubernetes Marketplace for Azure Stack is consistent with Microsoft’s Azure public cloud, as the template is generated by the Azure Container Service Engine. The resulting cluster will run the same containers as in Azure Kubernetes Service (AKS) and will use the Azure Resource Manager template to deploy and provision the resources for a standalone Kubernetes cluster.

Microsoft Azure Stack is only available on Microsoft-certified hardware stacks from Microsoft’s integrated systems partners Avanade, Cisco, Dell EMC, HPE, Huawei, Lenovo and Terra. We have heard that Microsoft, and its integrated systems partners are not selling a lot of Azure Stack yet. There are few customer stories and we believe that little hardware is being purchased from these partners.

Microsoft Azure is also a sponsor of Cloud Native Application Bundles (CNAB), a new cloud-agnostic open source package format specification created in close partnership with Docker and broadly supported by HashiCorp and Bitnami. With Cloud Native Application Bundles, cloud operators can manage distributed applications using a single installable file, reliably provision application resources in different environments and easily manage application lifecycles without having to use multiple tools.

Google Cloud’s Google Kubernetes Engine (GKE) On-Prem is in “early access” alpha release. Early access means that it is only available to a closed group of testers. Google Cloud says that GKE On-Prem will deliver provide a consistent GKE experience across Google Cloud and on prem datacenters. GKE On-Prem will enable installation and upgrades of Kubernetes in a local datacenter and provide access to Google Cloud Platform (GCP) services and cloud services exposed through the Open Service Broker API, supported by Google, IBM, Red Hat, SAP and others.

Google Cloud says that GKE On-Prem will install on customers’ existing on prem servers.

A serverless add-on for Google Kubernetes Engine is also in early access release. The serverless add-on will enable operators to deploy serverless workloads into a GKE cluster. It is not clear if this feature will be offered for GKE On-Prem.

AWS announced AWS Outposts at its re:Invent event in late November. AWS designed Outposts to provide an AWS experience with AWS developed hardware deployed in customer datacenters. We believe that AWS Outposts is designed to directly compete with Microsoft Azure Stack, above. AWS’s approach of using its own hardware vs. partnering with an OEM shows its willingness to compete directly with branded server vendors for enterprise IT share of wallet.

AWS Outposts custom on prem hardware will be sold by and fully managed, updated, maintained and supported by AWS. The custom hardware will be based on designs close to what AWS deploys in its public cloud. Customers will be able to order Outposts servers through the AWS Management Console. AWS says options will include ordering one or more servers, or quarter-, half- and full-rack units, with a range of compute and storage options. Outposts containers and VMs will be run on IaaS instances, identical to AWS public cloud behavior.

AWS plans to ship two variants of Outposts:

• AWS native variant will use the exact APIs and control plane as available in AWS cloud.
• VMware Cloud on AWS Outposts will use the same VMware control plane and APIs used in on prem VMware Cloud on prem and in AWS public cloud deployments (see more below).

AWS claims that Outposts will be available in the second half of 2019. We observe that Microsoft took almost four years to wedge a subset of Azure into Azure Stack. It is also very unusual for AWS to announce services that are not even in preview; AWS Outposts appears to be a year in advance. We are skeptical that AWS will be able to launch a production version of Outposts product before the end of 2019.

Cloud Native

IBM Cloud Private (ICP) is an application platform based on Docker and Kubernetes. It runs on IBM’s Cloud Managed Services (CMS) managed private cloud framework as well as any generic IaaS infrastructure, including IBM Cloud and other public clouds.

ICP can be used with IBM Multicloud Manager, released in October, to enable a hybrid container capability across IBM supported public clouds. IBM currently supports OpenStack, Red Hat OpenShift and AWS. IBM has committed to add support for Microsoft Azure by the end of 2018 .

IBM has published its minimum hardware requirements for running ICP in evaluation and production environments. ICP is not tied to purchasing IBM servers or other infrastructure. ICP is certified to run on IaaS instances of VMware vCenter, ESX, OpenStack and KVM hypervisors.

If Red Hat’s shareholders approve IBM’s acquisition, then IBM will have to decide whether to 1) keep investing in OpenShift, and by extension OpenStack, 2) keep investing in IBM Cloud Private’s cloud native approach, or 3) divide its attention between the two. We believe that investing for the long-term with IBM Cloud Private’s cloud native architecture is the better direction.

Oracle Cloud Native Framework (OCNF for convenience) launched during AWS re:Invent. OCNF is a freely downloadable package of open source software. It includes Docker and Kubernetes, plus the newly announced Oracle Functions serverless capability, based on the Fn Project. Applications developed for OCNF will be portable to any Kubernetes conformant environment and underlying hardware infrastructure. OCNF can be run in public clouds or on prem hardware, including but explicitly not limited to Oracle Cloud Infrastructure (OCI) public cloud and OCI at Customer private cloud hardware.

Oracle includes the following open source software in its OCNF stack:

• OS: Oracle already supports its own Linux distribution, OCI is based on it
• Development: Jenkins X, Helm, MySQL, Kafka, Spark, Spinnaker, Bitnami
• Observability: Prometheus, Grafana, Fluentd, Crashcart, Instana
• Provisioning: Terraform, Vagrant, Ansible, Chef, Puppet, Clair, Notary
• Orchestration: Kubernetes, Istio, Envoy
• Runtime: Fn, Docker, Kata, CRI-IO, Calico, Flannel, Ceph, Gluster, Java, Helidon.io, OpenJDK
• Security: Twistlock, Aqua

But that’s not all. Oracle has also integrated Kubernetes with Oracle Fusion Middleware (a WebLogic server) to support WebLogic Server applications in Kubernetes container environs.

We would also be remiss if we did not point out that Oracle still controls Java Standard Edition (SE) and includes Helidon.io in OCNF for Java SE applications delivery integration. That’s right, customers can run Java SE business logic from Kubernetes. For the more serious Java developers, we recommend paying attention to Eclipse Jakarta EE working group. Remember that Oracle donated Java Platform, Enterprise Edition (Java EE) to the Eclipse Foundation earlier this year. Jakarta EE is planned to be a cloud-native upgrade path for Oracle Java EE 8 (read more about Jakarta EE below).

Cisco Hybrid Solution for Kubernetes on AWS (CHSK on AWS for convenience) was also recently announced, and we didn’t see this one coming at all. Like OCNF, CHSK is cloud native: it does not depend on VM deployments of any type. CHSK runs on Cisco Container Platform on prem and AWS public cloud EKS, but it can also run in any on prem or public cloud IaaS environment.

Cisco is counting on a “better together” story with CHSK on AWS by integrating AWS’s Identity and Access Management (IAM) platform with Cisco’s Cloud Services Router (CSR) 1000v virtual router. CSR 1000v can run in a VM in any environment that supports VMs. CSR 1000v is designed to work better on Cisco HyperFlex/UCS hardware.

Unlike Oracle, Cisco’s minimum software-only subscriptions will start at about $65,000 per year for a typical entry-level configuration, in addition to any charges they run up in AWS for EKS clusters and Amazon Elastic Block Store volumes. While Cisco’s integration with AWS IAM is tempting, its pricing strategy will most likely limit potential customers to large enterprise IT shops that are already dedicated Cisco networking customers.

“Lift-and-Shift” Versus “Move-and-Improve” Versus . . . Java?

Moving VMs to the cloud using “lift-and-shift” means that IT shops need to rent large, well-provisioned instances that are roughly equivalent to the large, well-provisioned hyperconverged servers they’ve been buying for on prem use. The theory is that the low up-front investment in software porting and validation (i.e. none) will offset the higher operational costs of renting big, well-provisioned EC2 and S3 instances. Enterprise IT shops engaging in lift-and-shift hope that moving fast results in some kind of market advantage that offsets high operational costs over the remaining lifecycle of the application.

Lift-and-shift applies to migrating any application that is already running in VMware VMs, as well as applications running in any other portable VM, such as KVM.

The opposite strategy has been named “move-and-improve,” which means taking the time to refactor an application, rewrite it as cloud-native microservices, validate that the refactoring and rewriting process produces correct results (or perhaps the same wrong results as the original application). The idea is that a larger up-front investment results in buying the right sized instances to run a cloud-native application, saving a lot in operational costs over the lifecycle of the application.

Writing cloud-native applications to the CNAB specification is an example of move-and-improve.

There is another alternative, a middle ground. Java might present a way to lightly port applications from Java runtime environments to containers. Java is still one of the most popular business programming languages. The broader Java market includes both Oracle Java Platform Standard Edition (Java SE) and Enterprise Edition (Java EE).

Java SE applications can be run in open source Helidon.io wrappers inside containers. Helidon.io requires Java SE 8 or OpenJDK8 and the Apache Maven build environment. Also, AWS built its own implementation of OpenJDK8 and named it Corretto. AWS says Corretto is a drop-in replacement for Oracle Java SE. AWS is running Corretto internally to deliver production services, and it is in AWS customer preview now.

For Java EE fans, remember that Oracle donated Java EE to Eclipse earlier this year. Java EE has now become the open source Eclipse Jakarta EE working group. Jakarta EE will be designed as a cloud-native exact equivalent to Java EE 8. Jakarta EE will support implementing microservices and includes cloud native integration with Kubernetes. It will enable enterprise developers to lightly port Java EE applications to Kubernetes.

Eclipse Strategic members include CA (now Broadcom), IBM, Oracle, Red Hat and SAP. Eclipse Solutions members include Canonical, Docker, Google, Huawei, Intel, Microsoft and Salesforce.com. This is an impressive list of cloud supply chain and services vendors.

It looks like developers should be able to migrate a lot of Java SE and Java EE business applications to cost-efficient cloud-native environments without much fuss sometime in 2019.

Dell Technologies And VMware

In 2018, server hardware manufacturers HPE, IBM, Lenovo and Oracle all reaffirmed relationships with VMware. And yet, at the beginning of 2019, VMware is no longer at arms-length from Dell.

At the same time, AWS has become the public cloud target of preference for branded server providers to claim VMware lift-and-shift cloud capabilities. At the start of this year, Oracle was touting its ability to help customers lift-and-shift VMware VMs from its on prem infrastructure to AWS. Since then, AWS has decided to attack Oracle’s business from several angles, culminating in announcing Amazon Relational Database Service (RDS) on VMware at re:Invent as an Oracle database replacement.

VMware lists Cisco, Dell, Dell EMC, Fujitsu, Hitachi, HPE and Lenovo as Global Alliance partners. Cisco, HPE and Lenovo are direct competitors to Dell EMC in every server market, worldwide.

VMware announced its “Project Dimension” joint project with both Dell EMC and Lenovo in August at VMworld. Project Dimension runs VMware vSphere and vSAN, plus VeloCloud’s NSX SD-WAN on a purpose-built and validated hardware stack at customer on prem locations and bridges to VMware Cloud on AWS. Project Dimension is in beta now from VMware and both Dell EMC and Lenovo.

The heart of VMware Cloud on AWS (and AWS Outposts) is VMware’s Software defined datacenter (SDDC) software stack. SDDC is based on VMware Cloud Foundation: vSphere, vSAN, NSX and vCenter Server. VMWare’s existing customers can run VMware Cloud Foundation on their existing on prem server infrastructure, as they do now (presumably on the latest generation of a branded server OEM’s hyperconverged infrastructure). Customers can also upgrade to AWS Outpost’s custom designed hardware to run VMware Cloud on AWS on EC2 bare metal instances on-demand on prem.

VMware lists AWS, Google, IBM, Microsoft and Oracle as Global Strategic Technology Partners. However, the AWS and VMware joint announcement of VMware Cloud on AWS Outposts means that all of VMware’s public cloud relationships are no longer equal. AWS is now more equal than the rest when it comes to private cloud.

• Google GKE On-Prem and IBM ICP are software-only products, so they compete with VMware Cloud Foundation.
• Microsoft Azure Stack simply doesn’t support VMware VMs and Microsoft has no public roadmap to do so. Azure Stack has different use cases than VMware’s VMware Cloud Foundation.
• Oracle and AWS are directly competing for enterprise database customers. Dell/VMware is now a close strategic partner with AWS, not so with Oracle.

What To Expect In 2019

AWS usually announces availability of mature services at launch, even if the services are offered as a preview to start. Not so with AWS Outposts. While AWS is saying late 2019, it’s hard to see AWS having a much easier time than Microsoft had with Azure Stack. After its first attempt at fielding a private cloud, it took Microsoft nearly four years to ship a solid Azure Stack product. It will be difficult for AWS to carve out a robust subset of its public cloud to run in a dedicated hardware stack on prem and put the result into production in 2019. But AWS is not using “subset” language, it talked about Outpost offering general AWS public cloud services.

The importance of both public and private clouds puts Lenovo and HPE in a tough place. Both are partnering for all their critical software offerings. Meanwhile, competitor Dell is buying VMware, IBM is likely to buy Red Hat, Cisco, Huawei, and Oracle are writing and supporting their own open source distributions and both Huawei and Oracle are running their own public clouds.

HPE has backed off operating a public cloud and doesn’t seem to have a focus for private cloud. HPE looks focused on repatriating large enterprise customers to on prem virtualized infrastructure. HPE is selling a Microsoft Azure Stack solution (like most other server OEMs) but is mostly focused on its OneSphere multi-cloud manager and Greenlake Flex Capacity offering. HPE OneSphere supports VMware vSphere, AWS and Microsoft Azure public clouds and OpenStack APIs, which are not applicable to major public clouds outside of China. We recommend that HPE create a strong private cloud strategy and message, but we think it unlikely that HPE will acquire a private cloud in 2019.

Lenovo is a Red Hat partner and is also engaged with Microsoft Azure and Azure Stack. There are two directions for Lenovo to go: double-down with Microsoft Azure Stack and AWS+VMware, which has no differentiation from Dell strategy (plus Dell and VMWare are one and the same) and/or buy a private cloud distribution.

Canonical and SUSE are now strong acquisition candidates. Both SUSE and Canonical offer OpenStack and Kubernetes distributions. SUSE is owned by a private equity group and SUSE Linux Enterprise Server (SLES) is a more recognized name in enterprise. Canonical’s Ubuntu Long Term Support (LTS) distributions are widely used in the cloud, but Canonical is privately held and funded by founder Mark Shuttleworth. At some point, someone will make Shuttleworth an offer he can’t refuse for Canonical, but until then it will remain his company.

The two big announcements, Dell Technologies stock market re-listing leverage of VMware and IBM’s bid for Red Hat, will have a profound effect on the private cloud and multi-cloud landscape in 2019. It should be quite a ride!