Datacenter Networks Push Ethernet Switching To New Highs

The switching market has its ups and downs depending on the upgrade cycle for server processors and the nature of the economy at any given time. Despite the uncertainty in the economy, the Ethernet switching and routing markets keep humming along.

Ethernet switching is following its general trend up slightly up as it moves to the right, rising by 7.5 percent in the third quarter ended in September to just under $8.1 billion, according to the box counters at IDC. The router market in the datacenter, with these devices sold to both enterprises and service providers, was up by 4.7 percent in the period to just over $3.8 billion.

With regard to Ethernet switching, IDC says that sales of devices sold into the campus and branch office sectors did well, and so did sales of Ethernet switches to hyperscalers and cloud builders. In the quarter ended in September, the datacenter slice of the Ethernet switch market comprised only 12.9 percent of the total ports shipped in the third quarter, which we reckon came to 207.6 million ports worldwide based on what IDC said in its most recent report and what it has said in recent quarters; as best as we can figure, that comprised a total of 1.822.8 Eb/sec of aggregate switching capacity sold during the period. Do the math, and the datacenters of the world consumed some 26.8 million Ethernet ports in Q3 2021, up 8.7 percent year on year. The datacenter portion of the Ethernet market represented 56.7 percent of the nearly $8.1 billion in total Ethernet sales, which works out to a little more than $3.5 billion, an increase of 8.8 percent from the year earlier period. Our guess is that the datacenter represented a similarly large portion of the aggregate Eb/sec of switching bandwidth sold, too.

What Price Bandwidth?

We like watching technology trends here at The Next Platform, and we use the public data from IDC for servers and switches to try to get a proxy for capacity pricing in the markets at large on serving and switching.

In the past several quarters, we have seen the markets for 200 Gb/sec and 400 Gb/sec Ethernet switching take off, transitions that have taken longer than many expected but which are well under way. The hyperscalers and cloud builders are driving the bulk of this business, which increased 70.4 percent sequentially from Q2 2021 and port shipments rising by 2.2X sequentially. Our guess – and we have to guess because IDC did not provide this information – is that 200 Gb/sec and 400 Gb/sec devices drove just under $500 million in sales and accounted for somewhere around 311 exabytes per second (Eb/sec) of aggregate bandwidth across the devices sold.

As best as we can figure, sales of 100 Gb/sec Ethernet switches rose by 13.8 percent to just under $2 billion, and somewhere around 8.4 million ports were sold to drive that revenue, comprising 836.9 Eb/sec of aggregate bandwidth. Sales of 50 Gb/sec, 40 Gb/sec, and 25 Gb/sec devices continue to tail off, and while the cost per port is higher on faster ports, the cost per bit moved has normalized and synchronized for these slower chips, as you can see from this chart that we built:

As you can see, the cost per bit moved continues to drop for 1 Gb/sec and 10 Gb/sec Ethernet switches, but these are perfectly acceptable devices for many campus and edge use cases and, in the case of 10 Gb/sec devices, they are suitable for many datacenter use cases that are not bandwidth sensitive. If we do a little math on what IDC has told the public, it looks like there were a total of 23.3 million ports shipped into the datacenter, an increase of 8.7 percent compared to the year ago period, and the switch revenue generated by this rose by 8.8 percent to $3.6 billion. Ethernet devices sold into the campus and edge spaces accounted for 87.1 percent of ports shipped and 56.7 percent of revenues in Q3 2021, which is a healthy reminder that while datacenters are driving port bandwidth and port count density, there is a very long tail for these technologies as they move to the campus and then the edge. For instance, 1 Gb/sec switches accounted for around 167.4 million ports and around $2.9 billion in revenues in Q3 2021 – and this technology standard has been around since the summer of 1998, during the dot-com boom.

It is going to take a very dramatic change in the world to get all of those campus and edge networks to move to even 10 Gb/sec, and given the desire of many companies to sell us metaverse or omniverse experiences, it would seem that even 100 Gb/sec or 1 Tb/sec links will be insufficient. There is a profound impedance mismatch between what the networking industry can deliver in terms of switch and port bandwidth and what most of the enterprise world uses to link to datacenters where the applications run.

That there is more than ten times as much aggregate bandwidth in the Ethernet switch market overall also goes to show you just how much chatter there is between servers and their modular and sometimes microservices architectures to compose the pages for the applications that get sent down those skinny wires (and often fan out to routers supplying wireless links).

You can have a 1 Gb/sec port for the cost of a single entrée in a decent restaurant; a port running at 100 Gb/sec or higher costs on the order of the entire bill for four people at an excellent restaurant, including a few decent bottles of wine and some strong coffee after desert. (We haven’t got out much in the last two years, and it is starting to grate on us a little.) If you look at it on a cost per Gb/sec basis, the entire Ethernet switch market averaged around $4.44 per Gb/sec of bandwidth. Those 1 Gb/sec switches cost around $17.35 per Gb/sec, but 100 Gb/sec switches are down to $2.35 per Gb/sec, and the faster 200 Gb/sec switches are under $2.00 per Gb/sec and the 400 Gb/sec switches are approaching $1.00 per Gb/sec, averaging $1.60 per Gb/sec across the combined bandwidths as IDC is currently carving up the market.

It is hard to imagine it, but the advent of 800 Gb/sec, 1.6 Tb/sec, and 3.2 Tb/sec ports is going to keep pushing those costs down and down and down, and it won’t be long before the cost per Gb/sec of bandwidth will be on the order of maybe 25 cents to 30 cents. This relentless cost drop is the only way that the metaverse or omniverse can ever get built, and if the industry can’t deliver on it, then it won’t get built.

While the original design manufacturers have a pretty healthy slice of the server market, they still have a relatively tiny (but growing) piece of the Ethernet switch market, as IDC shows:

For some reason, neither Hewlett Packard Enterprise nor Juniper Networks made this chart, but we have them in the chart we built from the IDC data:

Cisco Systems, which has owned switching and routing for a long time but which has faced more than a decade of intense competition, had a 45.4 percent share of all switching in the third quarter, while Huawei Technology, the juggernaut IT gear supplier in China, had a 10.7 percent share of revenues. Arista Networks, the big upstart in North America, had a 7.3 percent share, followed by H3C’s 6.2 percent share, HPE’s 5.8 percent share, and Juniper’s 3.2 percent share. It is not clear why H3C and HPE are being counted separately – they are counted together by IDC when the market researcher talks about servers.