There is nothing quite like the open source community to demonstrate the principles of freedom, democracy, and meritocracy – and the difficulties of bringing those principles to bear and keeping them pure when money is involved.
Open source software is not just about having access to source code, but that is a kind of protection against tyranny if parts of the community, particularly corporate sponsors who cut the paychecks for a lot of the developers – either directly or indirectly – who create open source software, particularly the Linux kernel and the operating system that is stacked up around it in various distributions.
Quite a big stink is being made their week as Red Hat has made some major changes to the CentOS variant of its Enterprise Linux. And that is mostly because since its creation CentOS has been what amounts to a community supported variant of Red Hat Enterprise Linux that sits downstream from the RHEL development – meaning, it is rolled up from the source code after Red Hat is done – and Red Hat has reversed the polarity of the CentOS project it took over in 2014 and plans to move it upstream, as CentOS Stream, thus turning it into yet another development release like the Fedora project has been for many years and which also feeds into RHEL in some fashion. (Don’t even start thinking about how CoreOS Linux, which Red Hat acquired in January 2018 and which underpins its OpenShift Kubernetes container platform, fits into all of this.)
As the world’s largest company devoted to the development of commercial grade open source infrastructure software and arguably the only company that will ever be able to make this model work from a commercial standpoint at this scale, Red Hat can afford to have many different kinds of Linux that its employees contribute code to. The company rakes in somewhere north of $3 billion a year selling support contracts for such software, and has a vested interest in making sure the Linux operating system keeps getting more and better features added to it as well as support for successive generations of hardware. And to be fair, Red Hat does its share of this work and has since the company was founded decades ago. It is in this sense, though, that companies really are paying for Linux.
CentOS Stream was announced somewhat innocuously in September 2019, two months after IBM closed its landmark $34 billion acquisition of Red Hat. That timing might be coincidence, but maybe not. IBM has promised to keep a hand’s off approach to Red Hat, and is a just as likely that the Red Hat team is making this change all on its own as it is likely that Big Blue is coercing it.
CentOS Stream was designed to create a half-way point between the Fedora development release, which is changing like crazy all the time, and the commercial-grade Red Hat Enterprise Linux release, which changes on a regular, predictable, and relatively infrequent cadence of about twice a year. To be more precise, CentOS Stream is the code-base for the minor RHEL releases, and parts of RHEL development were actually moved into the CentOS project to get everyone collaborating. Which was good.
Either way, Red Hat has had a free to pay problem even before CentOS was created in the late 2000s to package up the RHEL code and offer community support – meaning, free and collaborative – to that alternate, “bug for bug compatible” release of RHEL. There is nothing in the Linux licenses that prevents the community from spinning up and self-supporting its own variant of RHEL and calling it something else, but with somewhere around half of RHEL customers actually using CentOS – meaning they are not paying for support and therefore not helping Red Hat create its variant of the Linux operating system and its elaborate stack of container and virtualization management layers as well as storage.
When Red Hat was founded in the wake of Linus Torvalds creating the Linux kernel in 1991, and even when Red Hat went public in 1999, there were not yet hyperscalers and cloud builders and HPC centers were just in the middle of ripping out Unix-based federated NUMA system clusters and replacing them with Linux clusters supporting Beowulf and MPI. It surely looked like created an enterprise-grade Linux operating system that felt like a proprietary operating system like Unix or Windows in terms of its fit and finishing and upgrade cycle was going to be a great business. While it has been a good business to be sure – Red Hat has been plagued by the free to pay problem, and is no doubt tired of it. While the company never said this when it took over the CentOS project back in 2014, the idea was no doubt to have a tighter link between those who paid for RHEL and those who didn’t by using CentOS, presumably greasing the skids to get customers to move from free CentOS to paid Red Hat subscriptions.
But this market has not panned out as planned. The big hyperscalers and cloud builders have long-since grabbed a Linux kernel and maybe even a distro and created their own Linuxes, but if they could get out of that business they probably would. Facebook, for instance, intercepts CentOS Stream for its own implementation of Linux, adding whatever goodies it needs downstream, rather than just adopting RHEL proper. Somewhere north of 50 percent of the servers in the world are running some form of Linux, and somewhere around half of these, we estimate, are running a homegrown Linux that will very likely never be replaced by CentOS, CentOS Stream, or RHEL, much less any other Linux variant. (Ubuntu Server from Canonical, based on the Debian variant of Linux, has its own large base, and SUSE Linux Enterprise Server has a pretty large base as well.) In addition, HPC centers have their own variants of Linux (think Tri-Labs Linux) or those created by their supercomputer vendors (think the Cray Linux Environment) as well as a healthy dose of CentOS.
Many of the organizations do special things with Linux, tweaks for their own use cases, and they cannot be done by companies like IBM through Red Hat easily – and certainly not at the “free” price of having developers participate in the community and then using community support.
You might be thinking that one could fix this free to pay problem by making support contracts cheaper. So, in theory, if Red Hat cut RHEL support contracts in half, it could possibly pull in all of the CentOS users, keep the RHEL users, and have the same revenue stream and probably not have to deal with much more in the way of support calls because the upper echelon, who use CentOS in their distributed systems, probably know as well or better how to support their systems and applications than Red Hat does.
In practice, to make that economic argument, one would have to calculate the cost of self-support – those developers at the hyperscalers, cloud builders, and HPC centers are not free, after all, and are generally highly paid professionals at that. If you fully burdened their costs, you might discover that self-support is much more expensive than RHEL support. And still, while being a good economic argument, it wouldn’t matter because for these customers, homegrown Linux support is about having expertise and being self-reliant, and even perhaps to create a competitive advantage with the core Linux operating system.
What is the value of that, and how does it offset the cost? The very fact that there has not been a true standard Linux distro tells you. High enough that people will budget people instead of outsourcing support to Red Hat, SUSE Linux, Canonical, or any other company in most cases of upper echelon users.
Organizations that control their own software control their own fates. That is the lesson of both HPC and hyperscale.
The middle ground between those companies who pay for RHEL support and who can’t self-support is where CentOS has played, and that is where the newly constituted Rocky Linux, created by CentOS co-founder Gregory Kurtzer this week and in honor of one of his CentOS partners, Rocky McGough, who passed away some time ago. (Lance Davis is the other CentOS founder.)
Kurtzer was the HPC systems architect and technical lead at Lawrence Berkeley National Laboratory and for several years was the corporate advisor at RStor, which initially tried to commercialize the Singularity Kubernetes container that Kurtzer created at Berkeley Lab. Singularity was eventually acquired by Sylabs, where Kurtzer was chief executive officer until April of this year. Since then, Kurtzer has been running the HPCng project, which he says is making the next generation HPC environment, and now is taking on the additional task of essentially re-creating the CentOS project that Red Hat has just said has no future. But Rocky Linux, which has over 750 contributors to the project already, certainly does because people like Kurtzer never really needed Red Hat support anyway. Or, more precisely, they don’t think they do.
Imagine if Red Hat just vanished tomorrow. That all of the programmers who work on Linux from Red Hat just stopped, that the testing and qualification process just stopped. The Rocky Linux community might just find out they needed Red Hat more than they reckoned.
What is clear is that Rocky Linux is dependent on all of the work Red Hat and the rest of the Linux community does, and it is equally unfair that users of Rocky Linux, who are among the smartest techies on the planet, are so unused to paying for support that they think it is free. It isn’t.
There needs to be a balance sheet that reckons value created by developers against costs to organizations for providing support. Or, we just keep creating free communities like Rocky Linux and pretending that it doesn’t cost anything.
Perhaps everybody – and we mean everybody – should always pay a little something, no matter how small per node, into the Linux Foundation to cover these support costs anyway. Perhaps the financing of the support model should be more like public radio (which doesn’t really work) and less like pay per view TV (which people sometimes hack to get around). Individual users might pay $50 or $100 per node per year or some such fee for community support, which would literally and directly underwrite the work Linux developers do the world over, and even Red Hat and Canonical and SUSE Linux would pay this per node. You could offset this with the value each company brings to the Linux stack. Do away with this notion that Linux is free. It really isn’t. It takes the life force of people to keep it going. If you want more handholding, then pay for a RHEL support contract from IBM.
It would be a messy balance sheet, to be sure. And one with a truly gigantic goodwill on that balance sheet, which is the most remarkable thing of all and something the Linux community should be proud of.