You might be thinking that with all of the investment in AI systems these days that the boom in InfiniBand interconnect sales would be eating into sales of high-end Ethernet interconnects in the datacenter. This is not the case.
According to the latest market research from IDC, there is enough buildout of 200 Gb/sec and 400 Gb/sec networks among hyperscalers, cloud builders, and some HPC centers and large enterprises that both the InfiniBand and Ethernet markets can grow at the same time.
Ethernet is, of course, used everywhere – edge, campus, and datacenter – unlike InfiniBand, which is used exclusively in datacenters. So it is important to make a distinction between sales of Ethernet switches into the datacenter and elsewhere. According to the statistics for the third quarter of 2023, the latest period for which data is available, IDC says that sales of Ethernet switches into the datacenter grew at 7.2 percent year on year, which is reasonable growth if not explosive, and reflects the fact that volumes are rising as prices are coming down across the spectrum of port speeds.
IDC did not provide a revenue figure for datacenter Ethernet switch sales this time around in its public statements about its Ethernet tracker, but we keep track of historical data from IDC and our math says datacenter Ethernet switch revenues hit $4.8 billion, which would be flat sequentially from Q2 2023 and representing 41 percent of all Ethernet switch sales. (If you want to know the exact figures, you have to buy the latest IDC report.)
IDC stopped giving out growth rates for ports by type and speed back in Q4 2022, which means we can’t calculate the ports shipped into the datacenter, and we have made estimates to fill in this gap. We openly admit that these are guesses based on the interplay of volumes and lowering prices as the supply chain is loosening in the datacenter Ethernet switch space. There is still a backlog, though, so prices are not going to collapse. In any event, our best guess is that 32.1 million ports were shipped into the datacenter in Q3 2023, which would represent a 20.3 percent increase in ports shipped year on year and a 24.5 percent increase sequentially from Q2 2023.
This is a far cry from the 5X increase in InfiniBand networking revenue that Nvidia had in its most recent quarter. For the trailing twelve months, we estimate in our model that Nvidia’s InfiniBand revenues rose by a factor of 3.2X to $5.53 billion. (This includes switches, network interfaces, DPUs, cables, and software, not just switches.) But the datacenter Ethernet switch market is still at an annualized run rate of around $20 billion, and if switching is about half of InfiniBand revenues, then datacenter Ethernet switching is still around 7X larger than InfiniBand switching – and the pressure is on to move more and more AI clusters to Ethernet and to get Ethernet technology on par with InfiniBand so companies don’t have to deploy anything but Ethernet except in the normal rare cases. We shall see how this plays out.
IDC says that in the non-datacenter parts of the Ethernet switch market, sales grew faster, rising 22.2 percent in the third quarter and up 36.5 percent for the first three quarters of the year as companies upgrade their campus networks and build out their edges. By our math, based on historical IDC data and growth trends, we think that the non-datacenter part of the market comprised $6.9 billion in sales and we estimate that just under 250 million ports shipped across a wide variety of speeds.
Add it all up and the Ethernet switch market across datacenter, campus, and edge came to $11.7 billion in Q3 2023, up 15.8 percent year on year. The companion Ethernet router market declined by 9.4 percent to just under $3.7 billion, and that is no surprise as routers are being increasingly built using merchant silicon that includes both switching and routing functions.
As we have been doing for years now, we try to sort out revenues and ports shipped by Ethernet switch speed, which used to be a lot easier when IDC gave us a little bit more data than it currently does in its public statements about its Ethernet tracker. (This is not a coincidence, obviously.) We have been estimating port counts since Q4 2022 and you should not think of our estimates as any more than informed guesses and hunches. Here is what the situation in Q3 2023 might look like based on our estimates:
And here is what the pricing trend lines might look like:
Here is what it interesting in all of this. Within the datacenter segment, sales of 200 Gb/sec and 400 Gb/sec Ethernet switches were up 44 percent year on year and port shipments were up 63.9 percent. Sales of 100 Gb/sec Ethernet switches rose by 6 percent in the datacenter as well as in the edge and campus.
IDC dropped this other tidbit: Original design manufacturers (ODMs) comprised 14.7 percent of datacenter sales, which works out to $705 million in the quarter, rising by 7.4 percent year on year from $656 million in the year ago period. In its commentary about the Ethernet switch vendors, IDC said that 72.2 percent of Ethernet switch revenues from Cisco Systems were outside of the datacenter, meaning that only 27.8 percent were inside of the datacenter, and based on Cisco’s overall 45.1 percent market share of the Ethernet market, we can calculate that Cisco had $1.48 billion in datacenter Ethernet switch sales in Q3 2023. Arista Networks, which is the main rival to Cisco in the datacenter aside from the ODM collective, had a 10.6 percent market share of Ethernet switching and 91.1 percent of its sales in Q3 2023 were into the datacenter, which works out to $1.14 billion. Cisco, Arista, and the ODMs accounted for $3.32 billion in datacenter Ethernet switch sales in Q3 2023, which is 69.2 percent of the market.
By the way, while Hewlett Packard Enterprise has a very fast growing Ethernet switch business thanks to its acquisition of Aruba Networks, only 8 percent of its sales, or around $73 million, was into the datacenter, according to IDC. We think that will change over time, and we will remind you that this number will also include sales of the “Rosetta” Slingshot variant of Ethernet that HPE sells for HPC and AI clusters.
Some observations.
First, Cisco has done a pretty good job protecting its datacenter switching business and expanding it despite intense competition, and is benefitting not just from a wave of campus upgrades, but also grew its datacenter business by 12.9 percent compared to 7.2 percent for the datacenter segment of Ethernet switching as a whole. We estimate that Arista’s datacenter switching business grew nearly three times faster than the overall market in the quarter, or about 20 percent. The ODMs are growing along with the pace of the market. If you take Cisco, Arista, and the ODMs out of the numbers, the rest of the market in aggregate had a 5.4 percent decline to $1.48 billion in sales.
Second, the server market outside of AI servers is in a recession, as we have pointed out many times, and the fact that Ethernet networks are still being upgraded, even at a modest pace, and considering still lingering supply chain constraints, shows how networking is rising in importance in the datacenter.
Great Ethernet, but What is about Ultra Ethernet?
Good year, all pals !!
Tim,
Thanks for the analysis, it’s always a joy to read what you present to us. Just a couple of comments.
1. I can’t believe that the avg cost per gigabit of 1gb network ports is *still* approximately the same as in 2017. And is so much higher than all the others. I would think that 1gb copper would be dirt cheap these days.
2. Can you *please* label your Y axis in your charts? I spent way way way too many seconds trying to understand the pricing chart showing avg cost/per gigabit. As my teachers always said, label your axis for credit!
3. Do these numbers include home networking? I’m sure 99% of most people are still just running 1gb networks, but there’s certainly a smattering of 2.5/5/10gb home networks. And couple of a insane people with 100gb or faster.
4. Man, you pay more per port, but the costs of running fast potrs is just insanely good these days. Now for a datacenter, you need to double the initial cost since I assume most people want redundant links once you’re at the 10gb/s or higher, and certainly for the 100gb/s speeds. So that’s a question of when do ports drop from dual-deploy down to single deploy at the edge? Interesting question as speeds go up, but management cheaps out and guys just enough to get the job done without any redundancy cause it’s too expensive.
5. Cisco is defending it’s turf because of all the network engineers who are trained in cisco, don’t want to change, hate the idea of a second vendor because they have to learn more, no one gets fired for buying cisco, etc. Me, I’m personally ticked at Cisco’s VMs which all fail horribly if you VMotion them while running. Kinda defeats the entire purpose of having VMs in the first place.
Man, this turned into a rant, time to chill in 2024. Happy New Year!
Up through Q3 2022, the cost per port was just math because IDC gave the numbers and all you had to do was calculate the percent growth year on year for revenue and ports and divide them.
I had not thought to worry if this included home networking–I assumed it was all corporate networking, not consumer networking.
Rants always welcome–they drive the thinking when they are a healthy rant, as this one is.
Thanks, John.
I hear you on the Cisco side. Their Meraki line is hanging on after the acquisition just because it’s too expensive and making too much margin to mess with. (Still a great product last I touched it though.) But the core operation is still roaming around waving tiny tyrannosaur arms making stuff that looks like the 90s when they were still cool.
Doubly feel your pain on the VM issues. Too much java, too many attempts to tie undisclosed “security” checks to physical hardware. They take an insane amount of time to boot/reboot as well, so if it has an allergic reaction it could literally be half an hour or more of down time.
We ran into an issue where if we had to reboot our core stack the Tomcat on the VM would fall over and then choke the service manager. So our minimum down time window went to like an hour after you add the still truly glacial cost of rebooting a stack of Catalyst series switches in the first place.