There are exceptions to every rule, but in general, when it comes to datacenter networks, enterprise customers are doing now what the hyperscalers and cloud builders were doing six or seven years ago. Each set of customers have very precise needs, which is why switch ASIC makers have different chips aimed at different parts of the market and why the major Ethernet switch makers – both OEMs and ODMs – have a wide portfolio of devices.
Enterprises have much smaller Ethernet networks linking their systems – by several orders of magnitude – and they tend to run at lower bandwidth networks as well – by maybe a factor of 2X to 20X, depending on the use case. It is tough to say if servers have lower Ethernet network interface speeds because they network has lower bandwidth or if the network speed drives the NIC speed. But what is definitely true is that enterprises are in different places in the evolution of their networks compared to the hyperscalers and cloud builders.
This set of charts from Dell’Oro Group showing the network interface port speeds on servers in the hyperscalers and cloud builders (on the left) and in the enterprise (on the right) illustrates the point:
Arista Networks, one of the major suppliers of Ethernet switching that addresses both the hyperscale/cloud and enterprise camps, has to be mindful of all of these differences and to resist the temptation to try to get enterprises to try to do what the hyperscalers are doing. Hardev Singh, director of datacenter product management at Arista Networks, walked us through the nuances of the market as we discussed some recent 400 Gb/sec and 800 Gb/sec product launches based on Broadcom’s “Trident 4” and “Tomahawk 4” families of switch ASICs that are aimed primarily at the high-end enterprise.
The Trident 4 family launched in June 2019 with a 25.6 Tb/sec device that could drive 64 ports at 400 Gb/sec. The Trident 4 line was expanded in December 2020 with denser 25.6 Tb/sec and 12.8 Tb/sec devices using PAM4 SerDes, and was updated with the Trident 4C in September 2022, which is a 12.8 Tb/sec ASIC that jacks up the packet inspection rate to a whopping 5.4 billion packets per second and a similarly whopping 500,000 simultaneous connections that can be monitored – that is five times the rate of competitive ASICs.
The Tomahawk 4 chips came out initially in December 2019 with a 25.6 Tb/sec that had 64 ports running at 400 Gb/sec using NRZ encoding on SerDes running at 50 Gb/sec, and was updated with 100 Gb/sec Serdes with PAM4 encoding in December 2020.
Generally speaking, here is how Singh says the hyperscalers and clouds are different from the enterprise customers:
The hyperscalers and cloud builders, says Singh, are running 200 Gb/sec and 400 Gb/sec ports – and often many of them – on server nodes running machine learning workloads, and using 800 Gb/sec interconnects between their datacenters, while enterprises are moving to 50 Gb/sec and 100 Gb/sec ports on their servers and moving to 400 Gb/sec ports at the core of their networks.
Even this is an oversimplification of what is going on, according to Singh, who breaks down the Ethernet networking needs of enterprises in this way by workload segmentation in compute, storage, and HPC/AI buckets:
As you might expect, HPC and AI workloads are commanding higher bandwidths on the network at enterprises just as they are hyperscalers and cloud builders, but unlike the latter, which have flat networks for both compute and storage, enterprises tend to push the bandwidth a little harder on storage than on compute.
Last year, Arista Networks rolled out switches in its 7050X4 lineup based on the 12.8 Tb/sec Trident 4 ASIC, and recently it is releasing switches based on the 8 Tb/sec version of the Trident 4 chip; both are aimed at 100 Gb/sec and 200 Gb/sec server ports, but can be used for customers who want to move up to 50 Gb/sec ports with cable splitters. Here are the variations in the new 7050X4 series:
The 7050 portfolio of machines is pretty broad and includes Trident ASICs running at 2 Tb/sec, 3.2 Tb/sec, 8 Tb/sec, and 12.8 Tb/sec speeds. The different ASICs throughputs provide different price points, and port for port at the same speed the new 8 Tb/sec switches will deliver a port that is 20 percent to 30 percent more expensive than those based on the 3.2 Tb/sec Trident 4 and 20 percent to 30 percent cheaper than the 12.8 Tb/sec Trident 4.
Similarly, last year Arista Networks launched higher-end 400 Gb/sec switches based on 25.6 Tb/sec Tomahawk 4 ASICs using 50 Gb/sec SerDes with PAM4 encoding that drove 64 ports, and it has recently delivered switches based on the 12.8 Tb/sec Tomahawk 4 using 100 Gb/sec SerDes with PAM4 that can drive 32 ports at 400 Gb/sec and a 25.6 Tb/sec Tomahawk 4 using 100 Gb/sec SerDes with PAM4 that can drive 32 ports at 800 Gb/sec. Here are the variations in the new 7060X5 series:
The faster 100 Gb/sec SerDes allows for Arista Networks to deliver bandwidth at 20 percent lower power and with cable splitters also allow the 800 Gb/sec devices to be split down to two by 400 Gb/sec and eight by 100 Gb/sec configurations for those who are not ready to upgrade their servers quite yet but who want to upgrade their networks now.
Here is a bit of advice as you think about your network upgrades. All PAM4 network interfaces can support NRZ mode to talk to switches that do not have PAM4 encoding. But network interfaces that are based on older NRZ encoding cannot talk to SerDes on the switches that have PAM4 encoding. So map out what NICs you need first because that will determine what switches you can talk to. If you can, it probably makes sense to just use PAM4 devices on both ends of the wire and just move into the future.
The new 7050X4 switches were in customer testing as 2022 came to a close, will be orderable early in the first quarter of 2023, and shipping in volume in the third quarter of this year. The two 32 port 800 Gb/sec 7060X5 switches based on the 26.6 Tb/sec Tomahawk 4 were also in customer testing at the end of last year and are orderable in the first quarter with shipments in the third quarter. The 32 port 400 Gb/sec 7060X5 switch is testing now, will be orderable in the second quarter, and shipping in the third quarter.