A New Era In Servers Is Starting Now

You know the world is a different place when shipping 2.58 million servers in a quarter feels like a slowdown, a disappointment, and perhaps a leading indicator of an overall economic slowdown in the world.

But the slight 5.1 percent downtick in server shipments in the first quarter of 2019 is probably more a function of the hyperscalers and cloud builders having spent a fortune on infrastructure in late 2017 and early 2018 and then waiting to see what the future holds for future processors from Intel and AMD.

In the quarter ended in March, according to the box counters at IDC, those 2.58 million server shipments drove a 4.4 percent increase in revenues, to $19.8 billion, and the fact that companies are buying increasingly rich configurations, with more main memory, lots of flash, and occasionally GPU or FPGA accelerators, is doing its part to keep the money pile growing even if it is possibly not doing as much as we might hope to increase the profitability of bending metal to build servers.

As far as we can tell, none of the OEMs or ODMs are rolling in black ink, much less swimming in it. But with the average selling price of a server falling through the 2000s, it has to have been a welcome thing to have it rising again through the 2010s. Here is what it looks like for just the X86 portion of the server space, based on IDC data, since the first quarter of 2009:

This is pretty much the mirror image of the decline in server average selling prices from 1999 through 2009 as proprietary and RISC/Unix systems fell out of favor – they used to represent half of system revenues in the mid-1990s, remember, and were the default iron for the dot-com boom until both X86 processors and Linux and Windows Server matured.

We start tracking the modern era of servers as the Great Recession was getting rolling in earnest in 2009. Coincidentally, that was when AMD was backing away from servers after some issues with its Opterons and Intel was, in essence, cloning many of the ideas of the Opterons in its “Nehalem” Xeons. That recession, plus the maturation of server virtualization, plus Intel’s delivery of a good 64-bit, multicore processor combined to put Intel’s Data Center Group on the path to a huge revenue expansion and a jump in profits that was bolstered significantly by the lack of competition.

There are many effects of the Great Recession on the systems business, but one of them is that it relegated all non-X86 – really non-Intel – processors to niche status. The RISC/Unix vendors just could not offer the same bang for the buck as a Xeon system running a virtualization stack and a Linux or Windows Server operating system. At the same time, the hyperscalers and cloud builders had an order of magnitude increase in their compute capacity needs as online applications and cloud computing became normal. So the relative capacity sold each quarter went through the roof, even as the core counts and instructions per clock cycle kept increasing with every processor generation and across every processor.

If you make some assumptions about how many cores were sold each quarter in the average server counted by IDC and layer in the IPC for the chips, you can come up with a relative aggregate performance capacity metric (in our case based on raw integer performance, excluding floating point math) for each quarter; we have the revenues from servers sold directly from IDC, and if you match these two lines up, it tells the story of just how much more capacity the world is consuming, relatively speaking, over the past decade. And it also shows how the price/performance increased dramatically as RISC/Unix machines were either vanquished from the market or dropped their prices to better compete.

A few things to note. What is immediately obvious is that every now and then, the market buys a lot of capacity and then cuts back a little bit. These events are not just tied to CPU product cycles, but they are related. What is also obvious is that the drop-off in capacity from Q4 2018 to Q1 2019 is the largest magnitude downdraft the server market has experienced in the past decade, but it really represents a return the normal slightly exponential but mostly linear capacity curve we have seen over the past decade. The sequential drop is, in fact, 4.4X larger than all of the server capacity sold worldwide in the first quarter of 2009, just to give that some perspective. The hyperscalers and cloud builders pulled back in the third and fourth quarters of last year, but enterprises were still pushing, and when enterprises and hyperscalers and cloud builders all pulled back in the first quarter, it caused that dip. But it is more like the hyperscaler and cloud builders were buying ahead of the launch curve, to be honest, and this is return to “normal.”

The thing is this: The world does a hell of a lot more computing than it used to as more and more aspects of our personal lives – and the corporations that ultimately support those lives – are computerized with all manner of telemetry and data.

The other neat bit about this chart is that it shows that even though the relative price/performance of the average raw server came down a lot between 2009 and 2011, it started petering out then and since Intel has had hegemony of the datacenter, the relative cost of a unit of raw compute in these servers has not come down. Sure, Intel is cramming a lot more raw integer in the machines, memory speeds are increasing to balance it, and flash is replacing disk to keep memory fed, and vector engine performance has grown even faster, but that raw underlying integer performance costs about the same today as it did three years ago. That is, however, an order of magnitude cheaper than it was when the Great Recession was roaring, so there is that. The curve is starting to bend down in the past two quarters, and with renewed competition from AMD, the core counts are going to go up and the prices are going to come down and set a new level a bit lower. And as far as we can tell, all of this really means profits will fall as those prices come down. The question with the future “Rome” Epyc processors from AMD and the “Ice Lake” Xeons from Intel is: Will the volumes expand fast enough so they can at least tread water with absolute profits even as the profit per server chip goes down?

Either way, there is little question in our minds that the days of 50 percent operating margins in server chips are over, unless AMD and Intel collude to keep pricing artificially high. Which is extremely unlikely and also highly illegal. Those profits will never come back, just like they didn’t when RISC/Unix jumped from the scientific workstation to the datacenter and lowered the bar competing with proprietary machines or when X86/Linux and X86/Windows jumped from the desktop and lowered the bar again against RISC/Unix iron. Arm is supposed to be spearheading then next bar lowering, jumping from our smartphones to the datacenter, but this has not quite happened yet. . . .

Money is how we keep score in business, and all we can see from the IDC numbers is what they reckon the factory revenues coming out of the OEMs and ODMs are; we have no idea from the IDC data if any of them are actually making money, and we suspect that in many cases, the OEMs and ODMs are bringing in a few pennies on the dollar for all their efforts and perhaps making more profits with services and sometimes software that they add to their machinery. It’s hard to say, even when looking at the financial results from public companies that play in the server racket. There’s a lot of nonsense in the numbers.

Here is how the vendors stacked up against each other in the first quarter of 2019 and the four quarters of 2018:

IBM, by the way, did not even make it to the top five vendors this time around as its System z14 upgrade cycle is already winding down and even despite growth in its Power Systems business. Selling off that X86 server business to Lenovo really did not help IBM’s systems cause, or its profitability, as it turns out. IBM is still not making as much money in systems as we think it thought it could, and for the reasons we suggest: Prices come down, and profits come down faster, when there is competition. This is just the way of the world.

Here is a longer view of the major OEMs plus the ODMs as a group since 2009:

Dell is the number one shipper of systems in the world, and the number one revenue generator, too, as it has been for quite some time now. Hewlett Packard Enterprise, including its H3C partnership in China, ranks second, as it has since Dell bypassed it. Dell is still growing, and HPE is flat, but it is not clear that either is actually profiting much by selling servers. (Someone has to build the machines, so we are grateful for the job they do and we hope that they can squeeze some more operating income out of their respective businesses.)

In the first quarter, Inspur (including its Power Systems partnership with IBM in China) and Lenovo and Cisco Systems were tied for the third place ranking in IDC’s numbers. Inspur brought in $1.22 billion in sales, up 36.4 percent and considerably faster than the revenue growth in the market at large, while Lenovo brought in $1.14 billion, up 3.9 percent, and Cisco brought in $1.05 billion, up 6.9 percent. The ODMs as a group brought in $4.55 billion in sales, down 1 percent, and the rest of the market (which includes IBM’s sales in this case) accounted for $4.34 billion, up 2.7 percent.

The alligator jaws that is the distribution of revenues by X86 and non-X86 server platforms continues to add teeth as 2019 gets started, and it is hard to remember the time, in the jaws of the Great Recession in fact, when sales of X86 and non-X86 systems were equal. Look at how quickly it diverged as the cloud builders and hyperscalers started building out their multi-million server node fleets.

Sales of IBM mainframe and Power platforms, the smattering of Arm systems, and the collection of other remnants of days gone past (mostly Sparc and Itanium systems) has reached a steady state of sorts of around $1.5 billion a quarter, give or take $200 million or so. The non-X86 iron is dominated by IBM System z mainframes with a healthy share of Power-based iron, and Arm machinery is still noise in the data. That could change, but the odds are increasingly less likely with a resurgent AMD. Arm chips may end up in networking and other gear in the datacenter, surrounding the servers even more than they do today, without making a dent in the actual servers – unless Intel or AMD stumble with their future X86 sales. After all the work that has been done with Arm server chips and the software stack, it would be interesting and fun to see Arm take hold. But for now, companies seem to be more focused on how much of their fleets might be based on Epycs as opposed to Xeons, and then there is always RISC-V to consider as another upstart eventually in the server space.

Whatever happens, it surely won’t be boring. That’s all we know.

Sign up to our Newsletter

Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.
Subscribe now

5 Comments

  1. Fascinating analysis! Thanks TPM! Any chance you can share some of your underlying analyses (i.e. spreadsheets etc)?

    It would be great to look at how much of this spending/compute capacity represents “HPC” by whatever definition. You could look at imputed prices for Top500 systems over time, investment in Infiniband etc.

    Perhaps we could collaborate on something along these lines?

  2. I’d like to see a breakdown on the ARM based server market and what potential there is for RISC-V and the newly opened MIPS 32/64 bit ISA based designs will that will do to the ARM ISA market, server and other markets. The MIPS Open initiative is just getting started with the opening up of the MIPS 32/64 bit ISA on a royalty free basis.

    But the Large server systems market is still very much x86 based and likely to remain so what with AMD’s more affordable and feature packed Epyc offerings across 2 generations starting with the Zen2 based Epyc/Rome release in Q3 2019. Before AMD’s Zen, Intel’s offerings where so high priced that many where looking at ARM despite any performance deficiencies as it was relatively easier for any ARM based server systems to make up for performance deficiencies with a wider price difference via lower pricing. But for the x86 market AMD’s renewed Zen based competition has allowed the server clients some alternative x86 ISA offerings at a much lower price and at a much similar performance and out-performance on some workloads compared to Intel’s x86 offerings.

    So any ARM based server options lost some price/performance advantage that they had against x86(mostly Intel at that time) in the pre Zen server market place! And that also came with AMD’s Epyc/SP3 Motherboard Platforms offering way more Memory channels(8 per socket) and PCIe(128 lanes for 1P and 2P MB platforms) lanes for the lower price relative to Intel’s highly segmented/higher cost x86 CPU/MB ecosystem offerings that offered less MB features overall and Intel wanting to charge more according to MB features offered.

    There are some interesting Power9 based small business and home server systems offerings now starting to appear from Raptor Systems/others for any folks interested in a fully open source firmware/software stack on Power9. So there are some niche server markets out there that may also be interesting for ARM, RISC, MIPS, as well as Power9 servers for small businesses and even home usage.

    So a review of the major non x86 market players as well as what the open source/royalty free ISA market will have in store on RISC-V and MIPS based systems and how that will affect the over all RISC based Server market in the future. Power9 and the home/small business server market also with there now being some very affordable 4 and 8 core power9 based systems offerings via OpenPower licensing.

    Personally I think that AMD can still derive a Higher ASP from its first generation Zen Epyc/Naples line of offerings simply by AMD offering some more Higher Clocked first generation Epyc/Naples variants. And like that higher clocked 16 core Epyc/Naples 7371 SKU that was released in Nov 2018 and maybe some higher clocked Epyc/Naples 24/32 core variants also on GF’s(Licensed from Samsung 14nm process node). The ASP gains for the 7371 and some lower tiered market server segment for AMD and a still useful Zen Epyc/Naples line even when Epyc/Rome reaches volume production. I’m sure that the binning has improved enough on that mature 14nm/GF-Samsung node for there to be available a nice full segment of higher clocked Epyc/Naples offerings for workloads that benefit more with higher clock rates on CPUs.

  3. Inspur achieved 6.2% good enough for 3rd place in the 2019 server market share on IDC’s report, dated June 5, 2019. They sold $1.22 B vs Lenovo’s $1.1 B.

  4. Q, TPM, how many processors in commercial DC server, 2P I’ve seen at OCP and elsewhere, then throughout one rack is the key question? Contained within that rack’s power envelope? Range of one rack power envelope? Depends on white space? High density city location v sprawling suburban. Please advise.

    How about providing a breakdown by sub system / component power budget in rack.

    Note; commercial analyst’s do not capture total Xeon volumes across server, workstation, Extremes, network/route/switch and communications control plane.

    Answer to the Intel processor sweet spot question –

    Intel x86 processor (cores) sweet spot on total Intel production volume;

    2600 v3

    Note 1; traditional Intel long run volume discount = 50% off 1K
    Note 2; cloud commercial volume Intel as price taker; 70% dealer up to 85% off 1K

    4C = 10.04%
    6C = 25.15% sweet spot 1, 1K AWP $544.61 / 6c = $90.77 per core
    8C = 18.91% midpoint, 1K AWP $1012.58 / 8c = $126.57 per core
    10C = 13.31% sweet spot 2A, 1K AWP $1466.38 / 10c = $146.64 per core
    12C = 11.52% sweet spot 2B for $1.74 more, 1K AWP $1708.48/12 = $148.37
    14C = 9.81%
    16C = 8.35%
    18C = 2.91%
    20C = 4.41%
    22C = 3.81%

    E5 46xx v3 4-way as a leading indicator;

    12C @ 29.60% of full run, 1K AWP $2838.21 / 12c = $236.51
    10C @ 27.99% of full run, 1K AWP $1658.01 / 10c = $165.80 sweet spot 1
    14C @ 19.31% of full run, 1K AWP $4727 / 14c = $337.64

    16C @ 25.35% of full run, 1K AWP $4727 / 16c = $295.44
    14C @ 18.46% of full run, 1K AWP $3827.27 / 14c = $273.38
    10C @ 17.32% of full run, 1K AWP $1816.24 / 10c = $181.62 = sweet spot 1

    E7 48xx v3 4-way as a leading indicator;

    8C = 28.50% of full run, $152.87 per core
    10C = 28.7% of full run, $150.20 per core
    14C = 26.31% of full run, $180.83 per core
    12C = 16.49% of full run, $214.50 per core

    E7 88xx v3 8-way as a leading indicator;

    18C = 36.18% of full run, $348.91 per core step up from sweet spot
    16C = 22.20% of full run, $275.83 per core sweet spot

    2600 v4

    4C = 11.29%
    6C = 11.10%
    8C = 18.23% step up from v3 6C sweet spot 1A, 1K AWP $661.01 / 8c = $82.63
    10C = 14.09% step up from v3 6C sweet spot 1B, 1K AWP $789.93 / 10c = $78.99
    12C = 7.35% midpoint step up from v3 8C, 1K AWP $1367.29 / 12c = $113.94
    14C = 15.99% sweet spot 2A, 1K AWP $1740.16 / 14c = $124.30 per core
    16C = 4.41% sweet spot 2B for $1.24 more, 1K AWP $2008.65 / 16c = $125.54
    18C = 9.32%
    20C = 4.41%
    22C = 3.81%

    E5 46xx v4 4-way as a leading indicator;

    16C @ 25.35% of full run, 1K AWP $4727 / 16c = $295.44
    14C @ 18.46% of full run, 1K AWP $3827.27 / 14c = $273.38
    10C @ 17.32% of full run, 1K AWP $1816.24 / 10c = $181.62 = v3 upgrade sweet spot 1
    12C @ 16.38% of full run, 1K AWP $2837 / 12c = $236.42 = v3 upgrade sweet spot 2

    E7 48xx v4 4-way as a leading indicator;

    10C = 47.67% of full run, $150.20 per core upgrade from v3 8C
    8C = 34.11% of full run, $152.87 per core upgrade from v3 8C
    16C = 9.27% of full run, $187.68 per core upgrade from v3 14C
    14C = 8.94% of full run, $150.50 per core sweet spot upgrade from all v3

    E7 88xx v4 8-way as a leading indicator;

    18C = 25.76% of full run, $252.89 per core upgrade from v3 16C
    24C = 18.93% of full run, $294.40 per core upgrade from v3 16C
    22C = 7.82% of full run, $277.61 per core upgrade from v3 16C
    20C = 3.22% of full run, $240.64 per core sweet spot upgrade from v3 16C
    14C = 0.91% of full run, $230.33 per core upgrade deal from v3 4/10C development

    XSL is a development environment for persistent / in memory compute and high bandwidth interconnect for 25/50/100 GbE+ networking.

    High to low volume as % of total;

    8C = 16.60% of full run
    12C = 13.11% of full run
    16C = 9.39% of full run
    4C = 8.13% of full run
    18C = 7.95% of full run
    28C = 6.87% of full run
    20C = 6.82%
    14C = 6.77%
    10C = 6.17%
    24C = 5.74%
    6C = 5.67%
    26C = 4.16%
    22C = 2.62%

    45.54% per core performance
    43.29% core power per watt performance
    7.06% 2x DRAM (Apache OPTANE development environment) = 2,843,292 XSL units
    2.22% Omni-path appears cancelled
    1.92% 10 year life

    XSL 1K AWP by W/P/G/S/B core grade excludes E;

    28C = $10,734.67 / 28 = $383.38 per core
    26C = $7,516.93 / 26 = $289.11 per core
    24C = $5,637.70 / 24 = $234.90
    22C = $3,655.00 / 22 = $166.14 at full run average marginal cost high
    20C = $2,683,16 / 20 = $143.16 at full run average marginal cost midpoint
    18C = $3,341.21 / 18 = $185.62 per core
    16C = $2,816.83 / 16 = $176.05
    14C = $1,769.35 / 14 = $126.38 at full run average marginal cost low is the enterprise applications sweet spot
    12C = $2,434.11 / 12 = $202.84
    10C = $955.96 / 10 = $95.60 per core Extreme’s sweet spot
    8C = $1,474.76 / 8 = $184.34
    6C = $856.18 / 6 = $142.70 at full run average marginal cost midpoint
    4C Large Cache development = $2,219.31 / 4 = $554.83 per core

    Note 1; traditional Intel long run volume discount = 50% off 1K
    Note 2; cloud commercial volume Intel as price taker; 70% dealer up to 85% off 1K

    XCL 1K AWP by P/G/S/B core grade excludes Px2 dice in package;

    28C = $11.385.92 / 28 = $406.64 per core (in OPTANE bundle deal?)
    26C = $7,405.00 / 26 = $284.81 per core upgrade from XSL
    24C = $6,334.18 / 24 = $263.92 per core
    22C = $2,713.11 / 22 = $123.92 at full run Average Marginal Cost low
    20C = $2,484.15 / 20 = $124.21 per core AMC is XSL volume upgrade sweet spot
    18C = $2,260.61 / 18 = $125.59 per core XSL sweet spot
    16C = $1,8744.74 / 16 = $117.36 per core v3 16/18C, v4 16/18C upgrade
    14C = XSL n/a core grade whose time has past 18C and less moves to Core line
    12C = $719.70 / 12 = $59.97 enters average marginal cost of Extreme’s / LCC reclaim
    10C = $2,745.64 / 10 = $274.56 is a development grade
    8C = $1,426.58 / 8 = $178.32 per core
    6C = $213 / 6 = $35.50 suspect variable cost of production
    4C = $1,221 / 4 = $305.25 per core development grade

    Answer to the Epyc sweet spot question by volume;

    16C TR 29/19xx = 31.90%
    32C TR 29/19xx = 19.83%
    12C TR 29/19xx = 15.09%
    8C TR 19xx = 9.26%
    8C E 7251 = 4.53%
    24C TR 29xx = 3.12%
    32C E 7601 = 2.68%
    32C E 7551 = 2.49%
    32C E 7501 = 2.07%
    24C E 7451 = 1.96%
    16C E 7301 = 1.45%
    16C E 7351P = 1.02%
    16C E 7351 = 1.01%
    16C E 7281 = 0.98%
    24C E 7401P = 0.90%
    24C E 7401 = 0.85%
    32C E 7551P = 0.59%
    8C E 7261 = 0.22%
    16C E 7371 = 0.04%
    Total = 100% of all TR and E

    Mike Bruzzone, Camp Marketing

  5. Clarification,

    I was formatting and left 46xx v4 dangling as my guide in 46xx v3 composition.

    E5 46xx v3 4-way as a leading indicator;

    12C @ 29.60% of full run, 1K AWP $2838.21 / 12c = $236.51
    10C @ 27.99% of full run, 1K AWP $1658.01 / 10c = $165.80 sweet spot 1
    14C @ 19.31% of full run, 1K AWP $4727 / 14c = $337.64

    This is v4 shown in v3 dangling adds 12C in v4 section.

    16C @ 25.35% of full run, 1K AWP $4727 / 16c = $295.44
    14C @ 18.46% of full run, 1K AWP $3827.27 / 14c = $273.38
    10C @ 17.32% of full run, 1K AWP $1816.24 / 10c = $181.62 = sweet spot 1

    Sorry for any confusion. Its tough being your own editor. Really, its Intel or AMD any enterprise evolving so rapidly at 60%, 70%, 80%, 90% + whole product for that phase of development, on development, validation.

    Commercially if it makes money for the customer at that stage in development its okay on a next deliverable striving for perfection.

    That’s innovation.

    Mike Bruzzone, Camp Marketing

Leave a Reply

Your email address will not be published.


*


This site uses Akismet to reduce spam. Learn how your comment data is processed.