There Is More Than One Route To SDN Switching

Software defined networking means different things to a lot of people, and the strictest meaning and most open (in terms of open source) variant of SDN will not necessarily – or easily – vanquish quasi-proprietary variants of SDN from the datacenter. The expansion of the Application Centric Infrastructure (ACI) approach from switching juggernaut Cisco Systems is a case in point, and it shows that the company can count on a fairly large number of customers to invest in the company’s vision for SDN.

There is no question that Cisco is under pressure in the datacenter switching business as open switch vendors and a handful of Linux-derived network operating system makers try to pry switching software loose from its hardware and essentially recreate the agnosticism that X86 servers have with regards to operating systems and applications. But to its credit, as The Next Platform detailed in its analysis of Cisco’s financials last month, has more or less maintained its switching revenues at around $3.5 billion per quarter, give or take, for the past five years. If anything, the open switch movement is possibly keeping Cisco from growing and putting pressure on its profits, but ACI has enough SDN capability that customers are willing to not only stick with their familiar vendor, but to use an SDN approach that ties them as tightly to Cisco’s systems as mainframe applications are lashed to IBM big iron. And that, of course, is precisely the point. Cisco is adding value, tying it to its own iron, and commanding a premium for it.

If the price of ACI hardware and its on-chip software acceleration gets too far out of whack with open switches and network operating systems, then you should expect to see very large customers migrate away from Cisco. Facebook is certainly doing this with its Wedge top-of-rack and 6-Pack modular switches, which have been contributed to the Open Compute community. Google has been making its own switches for a decade now, and Amazon Web Services does, too, because as James Hamilton, vice president and distinguished engineer for AWS, put it last year, the network is the real pain point in the datacenter and networking costs are rising much faster than compute or storage costs. (This is partly why clouds charge a premium for networking services.) But there are tens of thousands of very large organizations that have no desire to design their own servers, much less their own switches, and until open switches are installed at hyperscale, HPC, cloud, and large enterprise sites with a significant volume, Cisco’s ACI approach makes the most sense for its top or bottom line, even if it does give competitors selling open hardware and open SDN software an easier target to attack.

The ACI variant of SDN cooked up by Cisco puts much of the functionality that allows it to provide networking polices for applications, no matter where they are on the network, into specialized ASICs code-named “Northstar” for leaf switches and “Alpine” for spine switches. The special SDN ASICs were paired with not with Cisco’s homegrown switch chips, but rather with merchant silicon – specifically the “Trident-II” chips from Broadcom, which was just acquired by Avago Technology a few weeks ago and which has the kind of market share (around 65 percent) of the datacenter switching ASIC market that Cisco enjoys in the switch market. Broadcom chips are effectively the X86 chip for switching, and Cisco’s Nexus line is employing them with adjunct chips to take on the upstarts who will pair Broadcom chips with open source software to try to make a dent in the Cisco installed base.

The Nexus 9000 modular switches that Cisco debuted in November 2013 with ACI functionality did not require that customers use the ACI features. In fact, they could run a stripped-down and much-improved variant of the NX-OS network operating system that Cisco created for its Nexus family of switches and completely ignore ACI, or upgrade to this SDN functionality at a later date.

The ACI-enabled Nexus 9000 switch has been available for a year and a half, with the Application Policy Infrastructure Controller (APIC) software that rides on top of those Northstar and Alpine ASICs being available for a little more than a year. Cisco now has 36 key partners that plug into the ACI stack, ranging from Hadoop distributors to disk array makers to operating system makers to various network application providers and system management and application deployment software vendors. Importantly for the networking giant, Cisco now has 2,655 customers using its Nexus 9000 core switches and of these, 585 customers have fired up the APIC software to provide SDN functionality on their networks. This is but a dent in the Cisco installed base, mind you, but it is a start on a product transition that will take many years, and one that Jacob Jensen, who was a director of product management for the Insieme Networks spinout (which was spun back in) that created ACI and who has that same role inside of Cisco, tells The Next Platform that these products are “breaking away.”

By putting the SDN functionality on a chip, Cisco gets to keep its switch revenues and operating margins from collapsing; APIC functionality adds somewhere between 15 percent and 25 percent to the cost of a Nexus 9000 switch, Cisco has told us in the past.


Moreover, because customers don’t have to buy a lot of SDN and virtual switching software from VMware, the total cost of ownership of for a Nexus-ACI combination can be 75 percent lower than an SDN stack that has network functionality for the control plane and data plane spread out across switches and servers. The upshot back when ACI was released is that Cisco could build an ACI-capable network, with SDN functionality that allowed network and security settings to follow applications and their virtual machines around the network, using 10 Gb/sec switching for the same cost as using a VMware stack but with only Gigabit Ethernet switching.

In a more recent set of comparisons that Cisco is touting at this week’s Cisco Live customer and partner event in its hometown of San Francisco, the company is talking a bit about a case study from Symantec, which is using ACI-capable switches in its datacenter, as well as a report from ZK Research that pits the VMware NSX virtual networking plus switches from Arista Networks against Nexus switches and the ACI stack with VMware vRealize cloud management and compute nodes across both sets of iron:


As you can see, the delta in the overall cost, once you put in cloud software and the servers, does down quite a bit, but at least according to ZK Research, the overall savings for a complete compute-network stack are well within the zone at which companies will think about one approach or another. In the IT world, you need more than a 20 percent advantage to go through the pain of change. In the case of SDN, it is all new, so the barrier to adoption might be a little bit lower, but having the cost advantage is important. VMware and a slew of open networking proponents no doubt have their own reports which show their economic advantages, and we are digging around for such data. (If you have insight, don’t be shy.)

The details on the Symantec case study were put out in a case study done by IDC, and Jensen says that Symantec was building a new datacenter and put ACI-capable Nexus 9000 core switches at the heart of its network and server infrastructure, which of course was based on Cisco’s Unified Computing System blade servers and their integrated and virtualized switching. Symantec took the first four months of this year to stand up this iron and the ACI tools, and plans to migrate its corporate applications to the new datacenter over the next two years.

Thus far, one of the big benefits of ACI is that it can reduce the application development cycle time from testing to deployment inside of Symantec from three to four months to somewhere between two to three weeks, according to Vince Spina, vice president of IT, Global Network Infrastructure and Data Center Services at Symantec. Symantec is spending $26.8 million (after discounts) on the new infrastructure on the datacenter, and based on IDC’s calculations it is saving on the order of $145.11 million in benefits to the company compared to its current datacenter. A lot of the benefits are soft ones, such as risk mitigation and business productivity boosts, but a little less than half of the benefits that Symantec is seeing from the new datacenter are due to IT infrastructure cost reductions and IT staff productivity improvements, says IDC. The upshot is that Symantec is going to be able to recoup its investment in the new server and switching infrastructure in about eleven months – and Symantec gets a network with 40 times more bandwidth on the backbone with radically more efficient network operations.

Not everyone is going to go with ACI, and Cisco knows that. For one thing, the ACI approach ties customers to Nexus 9000 core/spine switches, which are not cheap. That is why the company has been pushing the BGP-EVPN protocol for virtual networks so hard in recent months. We explained the new protocol in detail back in February, but what Border Gateway Protocol Ethernet Virtual Private Network does is make the VXLAN Layer 2 overlay for Layer 3 networks much more efficient, so it is not flooding the network with management traffic just to shape the actual workload traffic.


“The first group is the mass market for SDN, and this is where the majority of Cisco’s enterprise, government, and university customers are sitting,” explains Jensen, “and from an SDN perspective, what they are really looking for is a turnkey solution. And that is what application centric infrastructure is. It is an integrated, centrally managed system built from the ground up with policy-driven automation for networks. We sell customers the car, not all of the parts. This is the architecture that we lead with and we are going to continue to invest in this.”

However, very large service providers have approached Cisco and they want to build very large network fabrics and they want to manage them using BGP-EVPN. “They can now use the VXLAN overlay model and then combine that with a network controller of their choice,” says Jensen.

In addition to just supporting the BGP-EVPN protocol on Nexus 9000 switches, Cisco is announcing a product called Virtual Topology System, or VTS, that makes use of the management APIs that Cisco has exposed in the NX-OS network operating system (in a reaction to OpenFlow and other open networking efforts) as well as the BGP-EVPN improvements for VXLAN to allow service providers to create their own programmable fabrics. Cisco is adding support for BGP-EVPN in the Nexus 5000 top-of-rack and Nexus 7000 end-of-row switches, and is promising to deliver VTS across its Nexus line, from the Nexus 2000 units in the UCS servers all the way up to the Nexus 9000 core switches, in the second half of this year. No word on what VTS will cost.


Cisco is also promising to make the APIs it has exposed in NX-OS running on the Nexus 9000 and its streamlined variant of the operating system available on the entire set of Nexus switches by the end of the year. Cisco is also announcing that it is integrating Microsoft’s Azure Pack, which is a set of tools for automating clouds that Microsoft sells in conjunction with Windows Server and System Center and which is derived from the tools it created to run its Azure public cloud, with ACI. This will ship in June.

Finally, Cisco expects that hyperscalers – what it calls mega scale datacenters – will just grab the NX-OS APIs and hook Cisco’s wares directly into their homegrown network management tools. Jensen says that Cisco is also allowing customization on those switches in the NX-OS – much as Intel allows hyperscalers to do customizations on their Xeon chips. Exactly what those customizations are remains to be seen. One thing for sure is that the updated NX-OS that will ship in the third quarter will have native integrations for Nagios, Ganglia, Puppet, Chef, and other DevOps tools commonly used among hyperscalers and cloud builders. Cisco will also have an SDK available for Nexus switch customers who want to build their own integrations.

To sum up the strategy, ACI is hardware-based SDN for those who don’t want to think about it, and Cisco is betting that a lot of the hyperscalers will settle for API access to NX-OS instead of having truly open source network operating systems. Cisco might be wrong about the latter, but if it is, the company was going to lose that business to the open switch upstarts anyway. Those remaining enterprise and public sector customers – calling what they might do is a tad bit harder.

It wouldn’t be a Cisco announcement without some hardware, and as part of the announcements this week Cisco is rolling out two new switches based on Broadcom’s “Tomahawk” ASICs. These two new switches are aimed at hyperscalers and cloud builders, and are meant to take on the Tomahawk-based switches that Dell announced back in April with very aggressive pricing. The full details are not available on these new switches since they are not shipping until the third quarter, but Cisco gave a few hints. The Nexus 3232C is a 1U rack switch that has 32 ports running at 100 Gb/sec and that has 3.2 Tb/sec of aggregate switching bandwidth; it uses QSFP28 cabling and will support port speeds of 10 Gb/sec, 25 Gb/sec, 40 Gb/sec, 50 Gb/sec, and 100 Gb/sec. The Nexus 3264Q has 64 ports running at 40 Gb/sec speeds and comes in a 2U form factor (it looks like two 1U switches stacked) and has 2.56 Tb/sec of switching bandwidth; it supports ports running at 10 Gb/sec or 40 Gb/sec. Both switches can be used as a leaf or spine in a flat network, and both have a starting list price of $35,000.

Drop an Alpine and Northstar ACI chip as an option on these babies, and Cisco might find that enterprise customers are a bit more enthusiastic about ACI. It would not be surprising at all to see Cisco do just that with the Nexus 3000s or the Nexus 5000s. The Nexus 9000 is overkill for many enterprises.

Sign up to our Newsletter

Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.

Subscribe now

Be the first to comment

Leave a Reply

Your email address will not be published.


This site uses Akismet to reduce spam. Learn how your comment data is processed.