The 400 Gb/sec Ethernet Upgrade Cycle Finally Begins

It has been a long time coming. Over two years, in fact. But the hyperscalers and cloud builders, and probably some of the backbones of telcos and other service providers, are finally getting prepared to make the jump to 400 Gb/sec Ethernet.

Arista Networks, the upstart that made a lot of noise in datacenter switching and put the hurt on Cisco Systems in a way that Juniper Networks really can’t anymore, has positioned itself with a move into routing and into campus networks as it bided its time on 400 Gb/sec switching, and is coming into its own in datacenter interconnects, or DCI, just as the 400 Gb/sec cycle is starting to move in earnest. We will see how much market share the company can eat and how much new ground it can break.

The indications are good, based on Arista Network’s financial results for the first quarter of 2021, its projections for the second quarter, and its visibility into the rest of 2021 and into early next year, the latter of which is both a blessing and a curse. The hyperscalers and large public cloud builders – what Arista Networks calls the “cloud titans” – normally do not give the company more than a one or two quarter window into their network plans.

But due to supply chain issues that are affecting every company that is a consumer of semiconductors and a manufacturer of devices, which is caused by parts shortages and labor shortages, the cloud titans and indeed other large organizations are giving companies like Arista Networks a longer view into their plans because it takes a longer time to fulfill those plans. Here is why: In some cases, there is a 52-week lead time to get components, and that’s because the coronavirus pandemic caused shortages of semiconductor substrates and silicon wafers as well as a reduction in chip assembly and packaging. Right now, in a lot of parts of the IT market, making the deals and signing the contracts is the easy part. Fulfilling them because of the shortages mentioned above is the hard part.

So it is important for companies to not let their ambitions get too far ahead of their capabilities. In a conference call with Wall Street analysts going over the numbers, Jayshree Ullal, president and chief executive officer, and Anshul Sadana, chief operating officer, said that Arista Networks was walking that fine line – particularly with regards to the 400 Gb/sec Ethernet rollout, which has been slowed in large part by the lack of affordable optics to plug into the switches and routers.

And as far as Ullal knows, customers are not pre-ordering lots of gear across multiple vendors to see what comes in, ready to cancel once they get something in with enough capacity to meet their needs. They are planning for delivery against what Arista Networks itself is planning, very carefully, to actually deliver when they say they can.

It didn’t help Arista Networks in that a number of vendors decided to skip the 200 Gb/sec upgrade cycle and waited to go straight to 400 Gb/sec. That certainly happened at Facebook, which was a material customers for Arista Networks during the 100 Gb/sec rollout and will probably be again in the 400 Gb/sec generation unless the social network decides to build all of its own switches and routers as well as complete network operating systems. If Facebook really did believe wholeheartedly in Open Compute, it would do just that. And the fact that it doesn’t just shows you how hard this networking issue really is.

The move to 400 Gb/sec Ethernet, which is being propelled by switch/router ASICs from Broadcom, Cisco, and Innovium, has a few drivers.

First, a switch with a single ASIC that can drive 400 Gb/sec ports is a lot less expensive than a collection of cross-coupled ASICs based on 100 Gb/sec or 200 Gb/sec pipes that are ganged up to give lots of ports. In many cases, six or more ASICs can be replaced with a single one, and it might cost half as much for that one piece of silicon as it did for the six pieces of prior silicon. So this savings is obvious and huge, and goes a long way to making datacenter-scale networks more affordable. Second, a 400 Gb/sec device – meaning one that drives 25.6 Gb/sec of aggregate bandwidth to handle 32 ports – can be converted to a much higher radix device, which flattens the network and also means fewer switches are needed to link the same number of devices. For many use cases, 100 Gb/sec coming out of the server is going to be enough for a long, long time. A 32-port 400 Gb/sec switch can be busted down to 128 100 Gb/sec ports, which is a rack of very dense two-socket servers in four-node enclosures (a common hyperscale and cloud form factor) plus some room left over for power distribution and switch bays. This port splitting drives the cost of a port way down – something the hyperscalers very much want. And if the same devices can be used with full-on 400 Gb/sec ports for DCI, well, that’s a win-win-win.

That is why the 400 Gb/sec ZR switch and optic standard, which is based on a PHY that can drive 80 kilometer ranges or higher – much further than the 10 kilometer range of the LR standard for 10 Gb/sec, 40 Gb/sec, and 100 Gb/sec interconnects and the optical transceivers that drive single-mode fiber using them. Arista Networks has done interoperability testing with 400 Gb/sec ZR optics on a testbed Microsoft set up, which places two datacenters 120 kilometers apart. This approach to CDI is direct and, as the chart above shows, eliminates a pair of transponders, mux/demux units, and line systems that are currently needed to do DCI. This basically makes a set of regional datacenters all look local.

Ironically, the 400 Gb/sec Ethernet delay has actually worked out for Arista Networks in that the company has spent the following couple of years acquiring Big Switch Networks and Awake for their respective network monitoring and security, using them to create a CloudVision stack that automates much of the operation and monitoring of networks from campus up through the datacenter with just enough machine learning to make it all work more easily.

In the quarter ended in March, product revenues rose by 31.2 percent to $539.1 million, the second highest quarter in the company’s history, which is pretty good for a first quarter and that is on par with the $555.1 million that the company booked in Q3 2019.

The services business, which literally means A-Car service switch and router support contracts plus other professional and consulting services revenues, came in at $128.4 million, up 14.5 percent year on year but off 1.4 percent sequentially. The recurring revenue stream of software and services together, which adds on licenses for EOS network operating systems plus sales of Big Switch and Awake software, came to $161.1 million in the quarter, which represented 24.1 percent of total revenues. We don’t have comparative figures for Q1 2020 because Arista Networks did not talk about this uniquely back then.

You can see the lack of enthusiasm of Facebook in the numbers, leaving Microsoft as the main driver of sales among the Super 7 – Alibaba, Amazon, Baidu, Facebook, Google, Microsoft, and Tencent. Arista only talks about these customers on an annual basis and only when they contribute at least 10 percent of its overall revenues – which is a practice set in place by the US Securities and Exchange Commission, not the particular generosity of any chief financial officer. Microsoft represented $498 million of revenues in 2020, down 10.1 percent, and Facebook represented $165 million of revenues, down 50.9 percent. There is the quadruple whammy of the 200 Gb/sec blahs, the 400 Gb/sec optics sticker shock, the hesitancy of buyers during the early phase of the coronavirus pandemic, and the consequent disruption to the supply chains of the world due to the pandemic. But it looks like the situation started to improve in the second half of 2020 and will continue to improve right up to some relatively tough compares for Arista Networks in the second half of 2021. The original plan was for 15 percent revenue growth in 2021, and now Ullal and Sadana say that Arista Networks can beat that. But they are not ready to say by how much.

Arista does not provide specific sector and product trends, but does offer some general observations about how sales workout each quarter at this point in its history:

A lot depends on the 400 Gb/sec rollout and the behavior of those cloud titans – and the service providers and lower tier clouds, and enterprises that will follow suit.

“The much talked about 400 Gb/sec upgrade cycle – or as we have been saying, the next-gen 100 Gb/sec, 200 Gb/sec, and 400 Gb/sec upgrade cycle – is expected to start second half of this year,” Sadana explained on the call. “The availability of ZR optics, MACsec encryption, and software features to monitor these optical links is also well-aligned in this timeframe. While our customers have always wanted multi-vendor environments, we remain their preferred partner and continue to get our fair share.”

How much share remains to be seen, particularly with Cisco putting out pretty decent switch and router chips with its Silicon One merchant ASIC lineup and an expectation to commercialize this chip internally in a wide and deep portfolio of products aimed at the same cloud titans.

“For quite some time, there’s been this talk about the 400 Gb/sec upgrade cycle and we have been saying for almost two years that it will take some time,” Sadana continued. “And while we always knew it would be coming, now we can see customers doing the pilot runs and so on and getting ready for second half. So that’s really what gives us the confidence. It’s not much more than that. There are plenty of issues on the supply chain side and so on that all customers have to work through as well. So, it’s really more in the planning stages right now. But we certainly are feeling better than before.”

Arista Networks better hurry before all of the cloud titans decide to do their own ASICs. At least one of the Super 7, Amazon Web Services, is reportedly thinking of making its own network ASICs.