Hyperscalers And Clouds Switch Up To High Bandwidth Ethernet

The hunger for more compute and storage capacity and for more bandwidth to shuffle and shuttle ever-increasing amounts of data is not insatiable among the hyperscalers and large cloud builders of the world. But their appetite for is certainly always rising and occasionally voracious – even when facing a global economy made uncertain by pandemic and trade wars as well as an actual war.

Nothing makes this more clear than the recent market research coming out of Dell’Oro Group and IDC, which both just finished casing the Ethernet datacenter switch market for the second quarter of 2022.

According to Dell’Oro, datacenter switching set a new record for revenues for any second quarter in history and also for any first six months of any year since it has been keeping records. Sameh Boujelbene, senior research director at Dell’Oro, was kind enough to share some insights on the quarter and forecast trends so we can share them with you. (But if you want to get all of the actual figures, you need to subscribe to Dell’Oro’s services, as is the case with IDC data, too.)

In the second quarter, according to Boujelbene, sales of switches into the datacenter grew by more than 20 percent. (Exactly how much, the company is not saying publicly. That strong double-digit growth is despite some pretty big supply chain issues that the major switch makers are having, notably Cisco Systems and Arista Networks, the two biggest suppliers in this part of the market. While the growth was broad across a bunch of vendors – meaning that most of them grew – only Arista Networks and the white box collective were able to grow their revenue share in the second quarter. Everyone else grew slower than the market and therefore lost market share.

That double-digit growth was across cloud service providers (what we call hyperscalers and cloud builders), telco service providers (what we call other service providers) and large enterprises, and each of these segments set revenue records for the first quarter and first half of 2022. All geographic regions, including China, where there were significant lockdowns during Q1, also showed growth.

At the high end of the datacenter switching market, with 200 Gb/sec and 400 Gb/sec devices, the number of ports running at these two speeds to combine to almost reach 2 million ports, which was more than 10 percent of all Ethernet ports shipped in Q2 and nearly 20 percent of all Ethernet switch port revenue in Q2. (The revenue split between 200 Gb/sec and 400 Gb/sec was about even, which means somewhere less than two-thirds of those 2 million ports were for 200 Gb/sec devices and somewhere slightly more than one-third of those ports were for 400 Gb/sec devices. The hyperscalers and cloud builders drove a lot of this demand.

“The forecast for 200 Gb/sec and 400 Gb/sec Ethernet is for these to more than double revenue in 2022,” says Boujelbene. “This growth is propelled by an accelerated adoption from Microsoft and Meta Platforms in addition to Google and Amazon. These latter two cloud service providers started adoption of these higher-speed Ethernet options way back in 2018 and 2019. Both 200 GE and 400 GE are forecast to more than double in revenue in 2022. The 200 Gb/sec and 400 Gb/sec Ethernet switch revenues comprised about 10 percent of the total Ethernet datacenter switch market in 2021 and is forecast to more than double in 2022, and in 2022 and 2023, we expect the 200 Gb/sec and 400 Gb/sec to continue to be driven mostly by the large Tier 1 as well as some select Tier 2 cloud service providers, while the rest of the market will remain predominantly on 100 Gb/sec Ethernet.”

The dropping cost per port is why the hyperscalers and cloud builders are moving to 200 Gb/sec and 400 Gb/sec Ethernet in their datacenter networks, but the absolute cost per port and the lack of need for super-high bandwidth among other organizations is why all but the biggest hyperscalers, cloud builders, service providers, and HPC centers will be sticking with 100 Gb/sec Ethernet – and sometimes even lower bandwidths.

One gating factor on the adoption of high bandwidth Ethernet is the cost of the optics used in conjunction with switches. And for the 200 Gb/sec and 400 Gb/sec generations, optics – meaning transceivers and cables together – account for two thirds of the network spending, while the switches themselves only account for one third of the overall cost. For the largest hyperscalers and cloud builders, the cost for the 400 Gb/sec optics alone was around $1,000 per port, or around $2.50 per Gb/sec, and that is expected to drop down to around $400 per port, or around $1.00 per Gb/sec. This pricing is not what other service providers or large enterprises or even HPC centers will pay. They all buy in much lower volumes, and will pay more per port and more per optics and cables.

Looking out across all of 2022, Boujelbene says that the datacenter switching market will surpass an annualized run rate of $17 billion as 2022 comes to a close, with around 70 million ports sold.

Here is the history and forecast, split by speeds, of port shipments by Dell’Oro from 2012 through 2026:

Right now, as you can see, most of the ports shipping into the datacenter run at 10 Gb/sec, 25 GB/sec, or 100 Gb/sec, with a smattering of 1 Gb/sec, 40 Gb/sec, and 200 Gb/sec or 400 Gb/sec. The demand for 10 Gb/sec switches is going to taper off pretty fast in the coming years, while 25 Gb/sec and 100 Gb/sec ports will hold relatively steady, 50 Gb/sec ports will find niches, and 200 Gb/sec and 400 Gb/sec and faster ports will grow.

“We expect 100 Gb/sec Ethernet to have a long tail,” says Boujelbene. “In fact, we expect 100 Gb/sec Ethernet to still comprise 30 percent of the switch ports by 2026. The deployment will be driven by different use cases. Currently 100 Gb/sec is mostly used in the fabric, but as prices continue to come down, 100 Gb/sec will also get adopted in the top of rack for server connectivity, which will drive a lot of volume. 10 Gb/sec died pretty quickly as it got replaced by 25 Gb/sec, which was priced almost at parity with 10 Gb/sec.”

Now, let’s take a look at the IDC data for all Ethernet switching, which includes datacenter, campus, and edge switching use cases. And then we can try to drill down into the datacenter a little based on what IDC has said in its public statements about its market research.

In the second quarter, IDC reckons that the world consumed just under $8.5 billion in Ethernet switches, an increase of 14.6 percent compared to Q2 2021. Router sales rose by 6.2 percent to $4.2 billion. If you do a lot of math on what IDC said in its statement, there were just under 32 million ports shipped that ran at 10 Gb/sec or faster speeds, up 25.4 percent year on year, and a total of 203.5 million ports across all speeds ranging from 1 Gb/sec all the way up to 400 Gb/sec, an increase of around one point year on year.

If you carve out datacenter switching from campus and edge switching (what IDC calls non-datacenter switching), here is the situation. In the second quarter, there was just under $3.9 billion in datacenter switching, up 27 percent, with port counts rising by 11.7 percent to 25.7 million. Most of the fast ports – those above 10 Gb/sec – end up in the datacenter, as you would expect. Non-datacenter revenue at the edge and in the campus accounts for a heck of a lot more ports at 177.8 million, down four-tenths of a point from Q2 2021, but even at low speeds and low costs per port, it all added up to $4.56 billion in sales, up 5.8 percent.

If you look it by vendor, here is the long-term trends we see in the IDC data across all Ethernet switching across the top four – Cisco Systems, Arista Networks, Huawei Technology, and Hewlett Packard Enterprise – plus Others:

You have to remember that Dell’Oro is only talking about datacenter switching and IDC is talking about all switching when they are talking about the Ethernet switch market by vendor. In the datacenter, only Arista Networks and Others grew market share but all vendors grew, and in the broader switch data from IDC, Cisco is the market leader with $3.57 billion in sales, up 10.2 percent, followed by Huawei with $895 million (up 9.4 percent) and Arista Networks with $856 million (up a stunning 55.1 percent). We know who is selling a lot of 200 Gb/sec and 400 Gb/sec switches to the hyperscalers and cloud builders now, don’t we? Also, in the quarter. 71.4 percent of its sales were for 100 Gb/sec switches, showing its strength at hyperscalers and cloud builders upgrading to 100 Gb/sec at the top of rack as Dell’Oro was talking about.

H3C had Ethernet switch sales of $532 million, up 4.8 percent year on year, while Juniper Networks fell 14.8 percent to around $220 million, based on our model of IDC’s data.

What we always find fascinating is how the cost per bit moved across switches goes down over time, with a kind of pulse that reflects market conditions in any given quarter:

It is amazing how expensive, on a per-bit basis, older technologies are. It is also amazing how many bits the newer switch ASICs can throw around each second, adding up to some pretty big money.