Puppet Takes Orchestration Up The Stack To Apps
October 5, 2015 Timothy Prickett Morgan
The mantra of the hyperscalers is “infrastructure as code,” meaning that all layers of hardware and systems software should be configurable through software, not by humans working manually through command line interfaces.
Configuration management tools like Puppet and Chef have come a long way in terms of giving large enterprises wrestling with scale tools akin to those the hyperscalers have created for themselves, and in some cases (like Facebook), the hyperscalers have turned around and put Puppet or Chef in their datacenter toolbox.
If you want to grow in this IT market, though, you have to always be looking up higher in the stack, higher value is as elements further down the stack commoditize. And that, Luke Kanies, the creator of the Puppet management tool and the CEO at Puppet Labs, is why his company is going one rung up the stack to cope with application configuration and orchestration.
“Since 2005, when I founded the company, in some ways we have been working on fairly straight forward problems, such as how do we configure these stupid computers, and in other ways we have tackled very sophisticated and difficult problems because it turns out that getting infrastructure to do what you want comes down to configuration and ongoing maintenance,” Kanies tells The Next Platform. “We want to help companies better cycle new technology in and old technology out, and we do not build software for the most leading edge companies in the world. Our goal is to help the majority of organizations out there get closer and closer to that leading edge. The goal is to shift the industry, not just individual organizations.”
The shift from manual system maintenance to DevOps is an important one, and one that not only makes administrators more successful at what they do, but makes them happy. They get interrupted less, and they get paged less. They stop wanting to quit their jobs after using Puppet. (And we presume after using other similar tools like Chef and Ansible, too.) They also can manage more infrastructure for the same pay, which makes their employers happy, too.
It is hard to pin down the precise numbers of companies that are using Puppet for system configuration management, but Kanies says that more than 10,000 people have gone through its training courses and estimates that 30,000 companies are using Puppet today; every day, another 22 join the fold, he says. Those are just the companies using the open source and freebie tools. In 2011, the company launched its commercial-grade Puppet Enterprise, and now has over 1,000 paying customers for this product, which are concentrated in the media, retail, financial services, and technology sectors with some presence from other industries such as manufacturing, distribution, government, and healthcare. As for the number of devices under control of Puppet, Kanies has not done that math in a while, but says he is “confident that it is in the millions of nodes,” with the largest commercial user managing 50,000 nodes. Kanies say that the average Puppet customer has somewhere between 100 and 1,000 nodes under management.
“No matter which way you look, there is no future where things get easier or less complex. We see a world with more servers and more services, but they are also more complex and more critical.”
Puppet Labs has grown fast alongside the popularity of its eponymous product, with over 400 employees now and 80 percent year-on-year revenue growth. (The exact figure is not being divulged.) The company has remained in its hometown of Portland, Oregon, and has brought in $87 million in funding in five venture capital rounds. Kanies says that Puppet is getting closer to going public, and is focusing on getting its cash flow positive, which should happen around the end of this year, and then its initial public offering will be gated by when it is a good time for Puppet Labs and what the conditions are going forward on Wall Street. Having Puppet Application Orchestration, as the new automation layer for applications is called, happen on this side of the IPO is a good thing, since it will only broaden the company’s appeal.
“This is the most significant and ambitious addition to Puppet Enterprise since it was introduced four and a half years ago,” says Kanies. “Puppet has been around for ten years, and this orchestrator is the major step for the next ten years. No matter which way you look, there is no future where things get easier or less complex. We see a world with more servers and more services, but they are also more complex and more critical.”
As an example, take a bank’s web site fifteen years ago. It was brochureware with bank logos, branch locations, phone numbers, and a few other bits of data. But today, the bank’s web site is, for all intents and purposes, the bank and the branch office is a relic of an earlier age. Moreover, cloud is no longer a future, but is the way modern infrastructure is built and services delivered, and while the DevOps movement – where the wall between application development and production environments is being smashed – is still nascent, it is growing. “Every CIO I know and every sysadmin I know is on board with the idea of agile operations and the criticality of aligning operations to the goals of the business.”
All of this is far easy when you have fairly homogeneous infrastructure and a relatively small number of applications that run on it, which the Googles and Facebooks of the world have. They have scale issues in terms of the size of the jobs they run and the amount of infrastructure it takes to run it, but they do not have the same level of scale that a large enterprise has when it comes to the diversity of the infrastructure and the workloads that run on it. One large customer that has been kicking the tires on the Application Orchestration feature for Puppet Enterprise has over 12,000 applications. It must take an army to manage this code.
That said, something has to manage the way applications behave, and Kanies says that according to data from Gartner, 85 percent enterprises do not use any automation for managing the configuration and deployment of infrastructure or applications. Millions and millions of people do these jobs today. “We don’t know if it is going to be in five years or fifteen years, but what we do know is that eventually this work is going to be done by software, not by people.”
So, it would seem that sysadmins can either be the Puppet or Chef expert, or they can be unemployed. But Kanies says this is not what actually happens in the majority of companies that deploy Puppet. While companies can use tools like Puppet to automate configuration management to reduce their costs, what they tend to do instead is have admins do more sophisticated things and increase their service levels. Application orchestration, for instance, will be something they pick up, if Puppet Labs has its way.
“If you are not planning to use Puppet Enterprise to manage your applications any time soon because the application engineers all work in a different building and dress differently and they don’t like you – a Jets versus Sharks kind of thing – the orchestration is still useful for infrastructure engineers because for infrastructure applications, ordering is just as important as it is for other applications.”
The Application Orchestration add-on to Puppet Enterprise uses the same modeling concepts as the infrastructure layer, which has over 3,500 modules describing infrastructure stacks – such as web servers or database servers, what have you. (That is three times the number of modules Puppet had two years ago.) Application Orchestration makes use of extensions to the Puppet DSL language used to create modules and these allow admins to model cross-host applications. Because the model is declarative, you don’t describe what you want the systems to do but the state you want the applications to be in and then Puppet figures out the best way to accomplish those goals as the application stack is rolled out onto the infrastructure. The relationships between the infrastructure and application software in that stack are sussed out by the Puppet tool itself and it can figure out the order in which to update the code when it needs to be changed. And, the modules can be stored, shared, and subjected to version control, just like any other piece of software.
Kanies says that Application Orchestration is being bundled in the Puppet Enterprise tool, rather than being sold as an add-on, because it will be immediately useful to infrastructure admins.
“If you are not planning to use Puppet Enterprise to manage your applications any time soon because the application engineers all work in a different building and dress differently and they don’t like you – a Jets versus Sharks kind of thing – the orchestration is still useful for infrastructure engineers because for infrastructure applications, ordering is just as important as it is for other applications. Even infrastructure applications should be modeled, and not just the individual pieces as if they are not unrelated. This update ordering functionality is valuable for everybody, as a result, we are not going to package it as an add-on product.”
The DSL language extensions that Puppet has made to create the Application Orchestration feature are being open sourced, but the first version of the full-on Application Orchestration tool will not be open sourced. Kanies says the framework is there for the open source version, but there was not time to get it done before this launch. A little further down the line, Puppet Labs will create a simplified version that will be open sourced and paired with the open version of the base Puppet tool. The Application Orchestration feature will be rolled out in the Puppet Enterprise 2015.3 update later this year, so presumably this open source variant will be let loose sometime in early 2016.
Puppet Enterprise costs $100 per node per year, with volume discounting to drive that price down. A node can be a physical server, a virtual machine, or a container.