Homegrown 25G Chips Help Cisco Hold The Switch Line

How much of a premium do you think that networking behemoth Cisco Systems plans to charge for its 25G Ethernet switches compared to their 10G predecessors? How does zero grab you? Or even less than zero?

That’s the plan, and this is surprising but competition will do that. Considering the profit margins that Cisco shoots for and has historically attained, you might be thinking that the company would be willing to walk away from deals, or at the very least switch to merchant silicon and get into a price war with its growing ranks of rivals in datacenter switching to protect its market share.

The company’s strategy is a bit more sophisticated than that, and so are its customers, as it turns out. Cisco has embraced merchant silicon chips from rival Broadcom to appease certain – and mostly hyperscaler – customers who have standardized on its “Trident-II” and “Trident-II+” chips for 10 Gb/sec and 40 Gb/sec products and who are adopting the “Tomahawk” follow-ons to them for 25 Gb/sec, 50 Gb/sec, and 100 Gb/sec gear. The Tomahawks are starting to ship from a number of switch makers, including Cisco. But don’t get the wrong idea. Cisco is still very much interested in etching its own switch chips, and this week is bringing out its own Cloud Scale family of switch ASICs that it says are better than the Broadcom Trident and Tomahawk alternatives.

Perhaps more surprising is that Cisco is going to be very aggressive on price – charging the same for 25G gear using its own chips that it charged for 10G switches also using its own chips and competitively priced against Broadcom-based machines.

“Quite frankly, the merchant silicon in the market today does not have all of the capabilities that customers need,” explains Thomas Scheibe, senior director of the Nexus line at Cisco, when asked why the company is still investing in ASICs. “We also wanted to have a chip that was on the latest 16 nanometer FinFET transistor technologies, which is used in the processors in servers, so we see no reason why we should not use it for switches.”

This process lead Scheibe gives Cisco a change to add flow tables, unified ports, congestion control, and other features to its chips that are missing from those of its rivals. Broadcom’s Trident-II chips, which were a central part of the 10 Gb/sec revolution in the datacenter, were made in 40 nanometer processes, but the first generation of Cisco ASICs used in the Nexus 9000 modular and fixed port switches were made using 28 nanometer manufacturing techniques. Similarly, the Broadcom Trident-II+ and Tomahawk chips are made with 28 nanometer techniques, and Broadcom is using the process shrink to add lots of features to those ASICs and to increase their performance, but with the Cloud Scale ASICs used in the second generation of Nexus 9000 gear, Cisco is jumping down to 16 nanometer processes and keeping a lead that, again, allows it to cram more features onto the chip.

“In the end, we go after the best price/performance, and we pick the best ASICs that we can,” Scheibe tells The Next Platform. “With the merchant silicon available today, you cannot build cost-optimized 25G switches with 48 ports and six uplinks running at 100 Gb/sec. The only other chip that can do this is Broadcom’s Tomahawk, but that delivers 3.2 Tb/sec of bandwidth and you only need 1.8 Tb/sec to support those ports. So the cost is too high. So we built our own ASIC to optimize the designs.”

That said, those customers who have standardized on Broadcom ASICs – which are akin to the X86 processor used in servers in terms of their popularity, although the Trident doesn’t quite have the ubiquity of the Xeon – are looking ahead to the Tomahawks, and Cisco will be shipping updated Nexus 3000 fixed port switches based on these ASICs later this year.

For the hyperscalers who create their own network operating systems and who have tuned them for the software development kits from Broadcom, making a shift to Cloud Scale chips would not be trivial not just because the chips are different, but also because the SDKs are different. The hyperscalers have long since automated the patching and provision of network software and changing this is very difficult for them. But not impossible.

The important thing for enterprise customers who are a bit less sophisticated on the automation front is that the Nexus switches all run the same NX-OS operating system, whether the machine is using a Broadcom or Cisco ASIC or whether it is a modular or fixed port switch.

Cisco has over 6,000 customers using its Nexus 9000 products, and more than 1,400 of them have opted for versions that use its hardware-assisted approach to software-defined networking, which is called Application Centric Infrastructure (ACI) and which is implemented in two ASICs code-named “Alpine” and “Northstar.”

We do not know what the code-names are for the three new Cloud Scale ASICs are, but we can tell you a little about them. One of the Cloud Scale chips is focused on the spine of the switch, where there is a need for high capacity and you want all of the switching and routing tables at full line rate, but maybe not all of the features such as policy enforcement or ACI that are necessary in the leaf switches. This big bad ASIC can support 36 ports running at 100 Gb/sec with its 3.6 Tb/sec of bandwidth; it can handle 5.5 billion packets per second. The second Cloud Scale ASIC has half the aggregate switching bandwidth at 1.8 Tb/sec, and because the core of the chip is half the size, it has room to add all of those other features like flow tables, unified ports. (We do not know the packet switching rate of this one.) A third Cloud Scale ASIC has 2.16 Tb/sec of switching bandwidth and can process 1.7 billion packets per second. In the Nexus 9500 series modular switches, the big bad chip runs in the fabric module and the smaller on runs in the line cards that slide into the chassis. In the fixed port Nexus 9200s (which do not have ACI) and Nexus 9300s (which do), the smaller ASIC again is used here.

The mantra at Cisco is 25G at the price of 10G and 100G at the price of 40G.

The Nexus 92160YC-X has 48 downlink ports running at 25 Gb/sec that can backstep to 10 Gb/sec when necessary (because the network interfaces on servers run at that speed) and have six ports that run at either 40 Gb/sec or four ports that run at 100 Gb/sec. It costs $20,000. The ACI-enabled variant of this switch is the Nexus 93180YC-EX, and it costs $22,500 and has all the same feeds and speeds otherwise. The currently shipping predecessor Nexus 9372, which has 48 downlink ports running at 10 Gb/sec and six uplinks running at 40 Gb/sec, costs the same $22,500. So if you don’t want ACI, it is cheaper to buy 25 Gb/sec ports than it is 10 Gb/sec ports, although Cisco won’t talk about it that way. We also suspect that Cisco will charge a slight premium for its own ASICs versus the Broadcom Tomahawks, but maybe not. Maybe its own chips will be significantly cheaper, and that is what chairman John Chambers was talking about last year when he said he would be taking the fight to the white box switch makers.

The Nexus 9500 series of modular switches are also being updated with the Cloud Scale chips. The backplane on this monster supports an aggregate of 172.8 Tb/sec of bandwidth on the largest configuration, which has sixteen line cards. (Other models have four or eight line cards.) Cisco is offering a 32-port line card with 100 Gb/sec ports based on the Cloud Scale chips, which can use cable splitters to drop the ports down to 25 Gb/sec or 50 Gb/sec. Fully loaded, the top-end 9516 can house 2,048 ports running at 25 Gb/sec, 1,024 ports running at 50 Gb/sec, and 512 ports running at 100 Gb/sec. Pricing for these modular switches is generally not given out, and this announcement is no exception. But it will be considerably higher than the cost of the fixed port switches based on the Cloud Scale chips – that much we know for sure.