Inside The Future Google Rackspace Power9 System

The OpenPower effort to create an alternative to the Xeon architecture in the datacenter just got a whole lot more real now that hyperscaler Google and cloud builder Rackspace Hosting have agreed to partner on a future server design based on IBM’s future Power9 processor that both companies intend to deploy in their datacenters.

Google, as one of the five founding members of the OpenPower Foundation in the summer of 2013, is always secretive about its server, storage, and switching platforms, absent the occasional glimpse that only whets the appetite for more disclosures. But at last year’s OpenPower Summit, Gordon McKean, senior director of server and storage systems design and the first chairman of the foundation, gave The Next Platform a glimpse into its thinking about Power-based systems, saying that the company was concerned about the difficulty of squeezing more performance out of systems, and his boss, Urs Hölzle, senior vice president of the technical infrastructure team, confirmed to us in a meeting at the Googleplex that Google would absolutely switch to a Power architecture for its systems, even for a single generation, if it could get a 20 percent price/performance advantage.

This morning, after a presentation at the OpenPower Summit, which is being co-hosted at the Nvidia GPU Technology Conference in San Jose, Maire Mahoney, engineering manager at Google and now a director of the OpenPower Foundation, confirmed to The Next Platform that Google does indeed have custom Power8 machines running in its datacenters and that developers can deploy key Google applications onto these platforms if they see fit. Mahoney was not at liberty to say how many Power-based machines are running in Google’s datacenters or what particular workloads were running in production (if any). What she did say is that Google “was all in” with its Power server development and echoed the comments of Hölzle that if the Power machines “give us the TCO then we will do it.”

To that end, Google announced a partnership with Rackspace Hosting to develop a server platform code-named “Zaius,” that will employ the future Power9 processor that IBM is expected to deliver sometime in the second half of next year. (We covered IBM’s update of the Power processor roadmap separately here.)

The Zaius machine will have two future Power9 server chips, which will have 24 cores and which will have both NVLink interconnects and PCI-Express 4.0 peripheral controllers, which will support IBM’s Coherent Accelerator Processor Interface (CAPI) overlay to hook accelerators and other devices directly in the Power processor complex and allowing them to share memory. The NVLink and CAPI interfaces in the Power9 processor will be the second generation of those technologies, presumably with lower latency and higher bandwidth. As you can see from the Google specs that Mahoney unveiled, the server will have 32 DDR4 memory slots, and with 64 GB memory sticks, that will yield a machine that will deliver 2 TB of main memory across those 48 cores. Interestingly, IBM, at the behest of the hyperscalers participating in the OpenPower effort, is dropping its “Centaur” memory buffer from the memory hierarchy with the Power9 chip aimed at scale-out architectures. This Power9 chip, known as the Power9 SO, short for “scale out,” is the one that Google and Rackspace want to put into future servers and is distinct from the Power9 SU, short for “scale up,” that IBM and perhaps others will use to create NUMA machines with more than two sockets. Google, as far as we know, is not interested in building big NUMA machines from Power9, but there is nothing that prevents it from doing so.

Google, as we have explained, will do anything to beat Moore’s Law, which it needs to do to continue to scale up and scale out its infrastructure, and that the appetite for more capacity for Google services such as Gmail, YouTube, and search were still unabated at the search engine giant, according to Mahoney.

“The demand for compute is growing and is relentless,” Mahoney explained in her keynote address at the OpenPower Summit. “We see that the energy efficiency gains that we used to get from making transistors smaller is declining, and an effect of that is that CPU top speed is leveling off and plateauing. We also recognize that the cost of making transistors smaller is increasing, and all of this overhead makes it more challenging for us to deliver on that equation of performance per TCO dollar. So we know that we need to have a different approach, and that is why Google is backing the vision that underpins the OpenPower Foundation.”

The Power8 chips got Google’s attention because of the massive memory and I/O bandwidth they have compared to Xeon processors, and it looks like Google and the other hyperscalers have been able to get IBM to forge the Power9 chip in their image, with more cores and even more I/O and memory bandwidth. “The vision is to build scale out server systems taking advantage of the amazing I/O subsystem that the OpenPower architecture delivers,” Mahoney added.

Aaron Sullivan, senior director and distinguished engineer at Rackspace who has been driving the company’s Open Compute server effort, and specifically its first generation “Barreleye” Power8 system, revealed some more of the feeds and speeds on the future Zaius Power9 system in his presentation and he added that Barreleye is being commercialized by a number of different vendors including Penguin Computing and Stack Velocity, was rolling into Rackspace’s own datacenters now, and would shortly be running various services on its cloudy infrastructure.

The Zaius motherboard that Google and Rackspace are co-designing looks like this:

The board is a little skinnier than the Barreleye motherboard, which means it will be able to fit in standard 19-inch racks instead of the 21-inch Open Rack designed by Facebook for its own datacenters and, while technically correct, has been by and large not adopted by the industry. There is a reason why Google and Facebook have just teamed up at the Open Compute Summit a month ago to create a new 48-volt open rack design that will come in the more standard 19-inch size. Google and Rackspace have committed to make the specs of the Zaius system available through the Open Compute effort, and it would not be surprising to see Google get more involved with other Open Compute projects over time.

We happen to think that Rackspace would have done something like Zaius on its own, but the fact that Google is helping with the design and presumably will deploy it in some reasonable volumes means that the ecosystem of manufacturing partners for the Zaius machines should be larger than for Barreleye. And with IBM shipping on the order of several tens of thousands of Power systems a year at this point, if Google and Rackspace dedicate even a small portion of their fleets to Power, it would be a big bump up in shipments.

There are a few other things that Sullivan divulged about Zaius. First, it has two NVLink ports exposed in the system. (The Power8 chips support six ports as far as we know, but the Power9 may or may not.) The Zaius system will have three PCI-Express 4.0 x16 slots, which is plenty for peripheral expansion, plus two PCI-Express 4.0 x8 slots and one Open Compute-compliant x16 slot. The Zaius board also has one M.2 flash memory stick port for local OS image storage, and interestingly a single – yes a single – SATA port for other storage peripherals.

Here is the conceptual layout of the Zaius sever as conceived by Sullivan:

The machine shown above is a follow-on design to the Barreleye machine that will fit into the 21-inch Open Rack, and it has a drawer with fifteen 2.5-inch SAS or SATA NVM-Express flash devices, which will hook into the server over PCI-Express ports. This is what Sullivan called the diskless option.

Google Code Stack Enabled On Power8

Mahoney said that a number of Google applications have been enabled on the Power servers that Google has designed and had built by its ODM partners. (It did not identify who that was.) And then Mahoney said this, which will certainly strike some fear into the heart of Intel, which has worked hard to keep hyperscalers like Google happy with its Xeon designs:

“We have ported our infrastructure onto the Power architecture. And most importantly, what that means is that our toolchain supports Power and so for our Google developers, enabling Power for their software applications is simply a matter of modifying a config file and off they go.”

This is not the first time that Google has ported its software stack to alternative platforms, of course, and McKean was very clear with us last year that Google is always testing out its software on alternative architectures to prevent “bit rot” and keep its options open. But what seems very clear now is that for certain workloads at least, Google is now in position to actually start rolling out its applications on Power. The fun bit about this is that we, as Google users, will never know if it does.