The high ground in the server market used to be large-scale SMP and NUMA machines with 16, 32, 64, or 128 sockets all lashed together to make a big shared memory machine. But that was back in the days when processors have one or maybe two cores, and the pace of Moore’s Law increases in transistor etching technologies has allows processor makers like Intel, IBM, and Oracle to cram a lot more cores and threads onto a single die. Enterprise workloads do not grow as fast as hyperscale and HPC parallel workloads, and that means over time a fairly modest machine from 2015 has the oomph of big iron from a decade ago.
Such is the case with machines based on Intel’s “Haswell-EX” Xeon E7 v3 processors, announced last week and scaling up to eight sockets using Intel’s on-chip NUMA links, and the Power8 midrange and high end, which IBM is updating to go up against Xeon-based machines, particularly for Linux-based applications. The Next Platform covered the expansion of IBM’s high-end Power E880 machines, which scale out to a maximum of 16 sockets and 16 TB of main memory, last week. IBM is formally announcing these high-end boxes at its Edge2015 conference in Las Vegas this week, but details on the largest of its Power8 machines slipped out a bit early. The final machine to be added to the Power8-based Power Systems lineup from Big Blue is the Power E850, and it is a four-socket machine that is aimed squarely at systems from Hewlett-Packard, Dell, Oracle, Fujitsu, NEC, and others that employ Intel’s Xeon E7-4800 v3 CPUs, which similarly support four-way NUMA clustering in their hardware.
The Power E850 is a bit different from four-socket boxes that IBM has shipped in the past. For one thing, it includes some capacity-on-demand features that up until now have only been available on larger Power Systems machines. With capacity on demand, IBM ships a box loaded with processors and main memory and allows customers to activate it as needed either permanently or temporarily on a daily or monthly basis with utility pricing. The base Power E850 system ships with two processors and a full memory complement (based on 16 GB, 32 GB, or 64 GB memory sticks) as a base, and customers active Power8 cores and memory in 1 GB increments.
The engines in the Power E850 are based on IBM’s “Murano” dual-chip module, which puts two half-cored Power8 chips into a single Power8 socket and links them by a crossbar. IBM uses dual-chip modules for a number of reasons, and the first is that a smaller chip has a high yield than a larger chip, generally speaking, and this is important because IBM’s 22 nanometer copper/SOI process, now controlled by GlobalFoundries, is nowhere near as high volume as Intel’s 22 nanometer Tri-Gate process, which is used to make the Haswell-EX Xeon E7 v3 processors. The DCM variants of the Power8 chips have more I/O capacity on their PCI-Express controllers, at 48 lanes per second instead of the 32 lanes per socket that are used in the single-chip variants of the Power8 chips used in the high-end Power E870 and Power E880 systems, which respectively scale to eight and sixteen sockets in a single image. These SCM Power8 variants do not need as much I/O bandwidth per socket because they have many more sockets in a system.
Initially, IBM will be supporting up to 2 TB of maximum main memory across the 32 memory slots, but Steve Sibley, director of worldwide product management for IBM’s Power Systems line, says that Big Blue will double it up again with its 128 GB memory modules, maybe later this year or early next. That will give the Power E850 the same maximum memory per socket as the top-end Power E880.
Intel supports 6 TB of memory across four sockets using 64 GB DDR3 or DDR4 memory right now, and IBM only supports DDR3 memory (which generally runs hotter for a given level of performance). The memory controllers in the Power8 chips were designed to be protocol agnostic and can support either DDR3 or DDR4 memory, but IBM tends to lag when it comes to memory because it likes to keep its memory costs low and its profit margins high.
Importantly, all of the enterprise-class Power8 machines make use of IBM’s “Centaur” memory buffer and controller chip, which has a chunk of 16 MB memory on it that is used to make up to 128 MB of L4 cache memory between the main memory and the L3 cache subsystem on the Power8 processors. The Xeon E7 v3 processors do not have L4 cache memory, and one of the reasons why IBM has been able to jack up the memory bandwidth on the Power8s relative to X86 architectures is this distributed L4 cache. Memory bandwidth and higher performance per core are the two key selling points that IBM is leveraging to promote the Power8 chip over Xeon alternatives. (No one talks much about AMD Opterons anymore, but that could change in a few years if AMD revamps its X86 server business as it plans to.)
As you can see, IBM is packing a lot of electronic components into the 4U chassis of the Power E850 system. The machine has four processors, and it is very likely that they run in a 190 watt thermal envelope (like the merchant silicon variants of the Power8 chips that Google, Tyan, and others are building systems based on) instead of the hotter 250 watt chips that IBM has used in its other and less densely packed systems. IBM is supporting three different variants of the Murano DCM: one with eight cores in the package that run at 3.72 GHz, one with ten cores that runs at 3.35 GHz, and one with twelve cores that run at 3.02 GHz.
The Power E850 has fans and drives in the front, and specifically, five large fans on top that blow first across the memory sticks, then the processors and then the PCI-Express slots in the back. Drive bays are below this – eight 2.5-inch drives, four 1.8-inch SSDs, and one DVD drive – and four power supplies fill the bottom of the rack behind the drives.
Customers with more storage requirements can hang up to four I/O drawers off the Power E850, which can have as many as 40 PCI-Express peripheral cards in them. The Power E850 system has eight PCI-Express 3.0 x16 slots and three PCI-Express 3.0 x8 slots internally. It seems very unlikely, given its dense packaging, that customers could put more than a couple GPU coprocessors into this machine, and given the workloads that the Power E850 is aimed at – database, in-memory processing (particularly SAP HANA and IBM DB2 Blu), analytics, and fat HPC cluster nodes – it is very unlikely that anyone will add GPUs to this machine. The Power E850 comes with a dual-port 10 Gb/sec Ethernet card in one of its PCI-Express slots by default.
Pushing Linux On Power Hard
Another thing that IBM is doing to keep the Power E850 competitive with systems using Intel’s Xeon E7 v3 processors (and the impending Xeon E5-4600 v3 variant of the Haswell chip, aimed at lower-cost four-socket machines) is offering what it calls Integrated Facility for Linux, or IFL, pricing on the Power8 cores in the Power E850 system. With the IFL approach, IT shops can restrict cores to running Linux and if they do, IBM gives them a big price break. The idea is somewhat counterintuitive, and the net result is IBM ends up charging customers using its own AIX Unix and IBM i proprietary operating system considerably more on processors and memory than it does for customers using Linux. This might be annoying to customers using AIX or IBM i, but they are a lot more captive (given that IBM is the only system supplier that supports them, and that probably will not change as the OpenPower systems come to market until IBM decides to get out of the server hardware design and manufacturing business) and hence have fewer options than Linux customers. IBM has to drop prices for Linux systems no matter what.
The Next Platform is putting together an analysis to compare the compute performance and price/performance for Xeon, Power, and Sparc processors to try to get a better handle on how these platforms stack up. Sibley says that a fully loaded, four-socket Power E850 with 48 cores will have somewhere between 5 percent and 10 percent more oomph in terms of raw performance compared to a four-socket Xeon E7-4800 v3 machine with 72 cores. Pricing for the machines configured with a hypervisor and Linux will be about the same, he says, and with a 70 percent utilization guarantee – meaning, IBM is promising that customers can load this machine up to that level of CPU capacity and still have workloads run with snappy response time – the gap widens up because, at least according to IBM’s tests, VMware ESXi on Xeons does not handle multiple workloads as well as IBM’s PowerVM hypervisor on Power8s. The gap on workloads could be as much as a 30 percent to 40 percent price/performance advantage favoring the Power E850 over a Xeon E7 box.
Intel can – and does – show its own charts illustrating how it beats Power8 machines.
What IBM is equally focused on is showing how the Power E850 offers a significant performance boost to its own customers running AIX and Linux workloads. (The Power E850 does not support the IBM i operating system, which is sure to annoy a bunch of the company’s customers who need more than a two-socket machine. The will be encouraged to buy a half-loaded Power E870 machine, which is more expensive and which is put into a higher software pricing tier, too, that will make IBM systems software and third party application software more expensive.)
The Power 750 four-socket machines that IBM announced in April 2010 using its Power7 processors are looking a little long in the tooth now and are the main targets in the IBM customer base where Big Blue and its partners are expected to push the new Power E850 four-socket box. Core for core, this Power E850 offers about twice the performance of the Power 750. The Power 750 used single-chip variants of the Power7 and Power7+ processors, but the Power 760 tested out the dual-chip module idea, and that is why you don’t see a Power E860 in the lineup. In effect, the Power E850 is the DCM variant; IBM did play around with the idea of having a SCM variant of the Power8 in a four-socket machine, but for whatever reason it has decided against the idea, so far. The point is, the resulting machine has a lot more oomph than the prior two generations, and customers who use this class of machine for application serving, database clustering, and in-memory processing will be looking hard at the Power E850. (Up until when the Chinese government put the brakes on buying Power Systems iron from IBM nearly two years ago, the Power 550 and Power 750 machines were hot sellers in the Middle Kingdom.)
The Power E850 will be available on June 5. IBM’s AIX 6.1 and 7.1 Unix variants are supported on the machine. The Power8 chip supports big endian byte ordering (used by prior Power chips) and little endian byte ordering (used by X86 processors), and now Linux variants can run in either mode. Red Hat Enterprise Linux 6.6 and 7.1 are both supported in big endian mode, and so is SUSE Linux Enterprise Server 11 SP3. In little ending mode, RHEL 7.1, SLES 12, and Canonical Ubuntu Server 14.04 and 15.04 are all supported. For Linux and AIX, IBM is also moving the machine down to the small software tier, rather than the medium one, which further lowers the price of application software on the box. This may or may not make software vendors happy. Pricing on the Power E850 was not available at press time, but we are digging.