Bringing Big Bandwidth To Large Enterprises

Hyperscalers change their datacenters – by which we mean whole generations of servers, storage, and switching – like regular enterprises upgrade server platforms. They change everything they are doing all at once and then beat the technology to death until it is time to upgrade the datacenter three or four or five years out.  Enterprises can’t do that, and they tend to make use of a wider variety of network protocols and also tend to keep their switching infrastructure around for anywhere from five to ten years.

But with 100 Gb/sec networking becoming more common in the enterprise – and necessary for supporting modern applications with a lot of east-west traffic across infrastructure within the datacenter – and 400 Gb/sec networking finding some use cases on the network backbone, the time is coming soon perhaps for large enterprises to make the jump from 10 Gb/sec and maybe 40 Gb/sec switching to 100 Gb/sec and realize the network performance enhancements that modern compute and storage require to balance out applications. That is, at least, what is driving switch ASIC maker Broadcom as it has started sampling its “Trident 4” StrataXGS BCM56880 family of chips.

If the “Tomahawk” chips are the thoroughbreds of hyperscale datacenter switching – minimalist, streamlined machines built for speed and efficiency – the Trident line is an older, workhorse line that has been aimed at datacenter networking in traditional enterprises. That means it has a lot of features, such as a broader array of supported protocols as well as advanced features such as load balancing, denial of service protection, and network address translation that enterprises need, but unlike the hyperscalers and their cloud builder peers they cannot program for themselves. The enterprise switches in the Trident family also tend of have bigger databases – access control lists and such – as well as deeper packet buffers.

While these feature differences (and their vastly different price points) have been the Trident and Tomahawk chips distinct from each other over the past decade, Broadcom is doing something interesting with the Trident 4 launch that could be repeated again, or not, according to Peter Del Vecchio,  who is the product line manager for the Tomahawk and Trident lines at the chip maker, and that is to make the upcoming Trident 4 pin compatible with the existing Tomahawk 3 chips, which launched in January 2018 and which delivered 12.8 Tb/sec of switching bandwidth and 128 ports running at 100 Gb/sec and 32 ports running at 400 Gb/sec. The 50 Gb/sec SerDes with PAM-4 encoding (which delivers two bits per signal and therefore 100 Gb/sec per lane) that came with the Tomahawk 3 and now with the Trident 4 are what allows those port speeds to jump and the price per bit transferred to drop. While the Tomahawk 3 chips were implemented in a 16 nanometer process from Taiwan Semiconductor Manufacturing Corp, as were the Trident 3 chips that came out in the summer of 2017, the Trident 4 chips are jumping to the 7 nanometer processes from TSMC that are becoming the standard for next generation CPUs, GPUs, FPGAs, and switch ASICs.

If you want to know all about the history of the Trident family, we got into that detail two years ago in our Trident 3 coverage, so we are not going to cover all of that ground here again.

By switching to 50 Gb/sec and PAM4 encoding, Broadcom is able to quadruple the bandwidth per lane and therefore the bandwidth per port in under two years between the jump from Trident 3 to Trident 4, and with the socket compatibility between Trident 4 and Tomahawk 3, switch makers can drop this enterprise-class ASIC into their hyperscale designs and have an enterprise-grade switch out the door licketysplit. That’s the idea anyway, but we expect more than a few switch makers will have unique designs for the Trident 4 ASICs when all is said and done.

While Broadcom is not providing pricing for the ASIC or what it thinks estimated street prices will be for switches based on the Trident 4 chips, Del Vecchio tells The Next Platform that compared to other ASICs that are delivering 100 Gb/sec and 400 Gb/sec ports out there today, the Trident 4 can cut the cost per port by 75 percent and also reduces power consumption, and therefore most of the operational cost of the switch, by 75 percent as well. Broadcom picked on the “Doppler” UAVP 3.0 chip from Cisco Systems, used in its very high end Catalyst 9600 switch/router hybrids, and here is how it stacks the two up against each other:

That Doppler chip delivers less that 2 Tb/sec of aggregate bandwidth, and to create a system capable of delivering 128 ports running at 100 Gb/sec it takes three Doppler ASICs to make the backplane in three different enclosures plus another and another nine Doppler chips to create the ports in the modular switches. You can do the whole shebang with one Trident 4 chip in a fixed port switch without having to go modular at all. The Trident 4 offers 3.3X the switch capacity – 5 billion packets per second for one Trident 4 versus under 1.5 billion packets per second for the Doppler chips from Cisco, and the Trident 4 has more than twice as many IPv6 routes in its chippery than the Doppler chip, too, at 620,000 routes. The Cisco chip is not open or programmable, as the NPL language for the Trident chips from Broadcom is. The NPL  programming language is open source as well as open in terms of APIs being fully described, as is the case with the P4 programming language from Barefoot Networks, which is in the process of being eaten by Intel.

By the way, the Trident 4 is not using a chiplet architecture, as the Tofino 2 chip from Barefoot does, putting chip cores on 7 nanometer processes but I/O and other functions of the chips on earlier 16 nanometer processes. Trident 4 is a monolithic chip etched in 7 nanometers and weighing in at 21 billion transistors total. This is the largest chip we have heard of to date, more dense in terms of transistors than even the “Volta” V100 GPU from Nvidia, which has 18.1 billion transistors on its die using 12 nanometer processes from TSMC.

At this point in the Trident family, the pipeline on the chip is fully programable, something that enterprises want so new protocols can be added to the switch without having to trade out the device. This is a change that the hyperscalers and cloud builders have been fomenting, and now everyone wants it. The difference is that the hyperscalers and cloud builders do a lot of their own network operating system development and therefore code their own protocol changes, whereas enterprises generally do not have this expertise. But the still have the need sometimes, and they will now be able to either encourage Broadcom or their switch makers to tweak or add protocols as necessary while at the same time prolonging the field life of their switches.

There is no set pattern to now this is done.

“We see some of the very large end user customers interested in adding proprietary security information that is just for their network to try to give them a bit more security, or they may add a whole new application,” explains Del Vecchio. “They can take that code back from Broadcom and make whatever tweaks necessary. One common thing that we see is custom VXLAN headers, where companies insert some proprietary security information and propagate it through their network. The bottom line is they add flexibility and also future proof all of their hardware. A lot of times, for very large customers, such changes are done by the OEMs or by Broadcom, but we certainly can enable the end customers to do it on their own. Some hyperscaler or telcos really want things to be completely proprietary and they don’t even want to tell us about what they are doing. In that case, they actually make the tweaks to the code.”

Customers can use the APIs in the SDK6 toolchain, as they probably did with the Trident 3 and Tomahawk 3 switches, to tweak the network operating system and features in the Trident 4 or they can use the SDKLT logical table and database approach and its APIs, which were open sourced a few years back by Broadcom as P4 was taking off and becoming a threat.

Broadcom is offering four different variants of the Trident 4 chips. The X3 chips offer 2.1 Tb/sec of switching bandwidth and are aimed at access point jobs in the edge and campus networks. The X4 variant if aimed at the distribution and aggregation layers in the edge and campus networks, and delivers 4.25 Tb/sec of bandwidth. The X9 variant of the Trident 4 is aimed at the converged core or datacenter spine layer of the datacenter network, and offers 8.5 Tb/sec of bandwidth, and the X11 variant offers 12.8 Tb/sec of oomph. By putting the same chip across all of these uses, the same chip architecture, Ethernet feature set, and network operating system can be deployed end to end in the enterprise network. And this is precisely how Broadcom wants to make it all up in volume as the number of switches needed to create fabrics is going down and customers are using bandwidth to increase witch radix instead of switch bandwidth.

We will see if it works out that way. Switch ports are not growing as fast as bandwidth at the moment, and that could mean a price war.

Sign up to our Newsletter

Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.

Subscribe now

1 Comment

  1. This is a pretty fascinating move by Broadcom in a couple ways. First off, toting out a monolithic 7nm programmable switch chip of this magnitude is kind of ridiculous, considering that none of the would-be competitors (Intel-Barefoot/Innovium/Marvell) have shown the technical chops (yet) to pull this off, even on simpler non-programmable silicon. For those who understand how hard this is, it seems Broadcom is really flexing it’s muscles in the presence of the competition. The speed at which they’ve executed, from Tomahawk 3 to Trident 4 to the upcoming Tomahawk 4, is very impressive. And making this new entrant pin-compatible with all the existing Tomahawk-3 reference systems has major implications on TTM. Very interesting.

    For those would-be challengers, this is starting to look like one of those nature videos where a lion chases a deer around for awhile, realizes the deer is too fast, and gives up to grouchily consume some grass, while pretending it wanted grass all along 🙂 If you are in the business of chasing deer, you better be faster than deer, or find another business to be in.

Leave a Reply

Your email address will not be published.


This site uses Akismet to reduce spam. Learn how your comment data is processed.