In one fell swoop, Intel has finally filled a giant hole in its switching product line by acquiring upstart Barefoot Networks, the creator of the P4 programming language for networking devices and the “Tofino” family of Ethernet switch ASICs that make use of it. Now, perhaps, Broadcom will finally and forcefully get the kind of competition in datacenter switching that upstarts Barefoot, Mellanox Technologies, and Innovium were trying to bring to bear by themselves.
With Nvidia buying Mellanox, which supplies both InfiniBand and Ethernet ASICs and switches, for $6.9 billion and Intel paying an undisclosed sum for Barefoot, that probably means someone – IBM? AMD? Xilinx? Ampere? Probably not Marvell since it has XPliant already? – will probably take a run at Innovium before too long.
Intel is, of course, no stranger to datacenter networking, but for some impenetrable reason, it has not pushed into Ethernet at the switch even though it offers Ethernet network interface cards for servers. We suspect that Intel believed that it could leverage InfiniBand for HPC and then make it more mainstream – the company was one of the standard bearers, along with IBM, of the technology way back in the 1990s – but the company has never explained why it did not aggressively sell and evolve the Ethernet switch chip lines of Fulcrum Microsystems, which Intel bought for an undisclosed sum in July 2011.
Fulcrum was founded in 1999, under the name Asynchronous Digital Design, by Uri Cummings and Andrew Lines, both of whom were researchers at the California Institute of Technology, and the idea was to commercialize the ideas they had developed with their professor, Alain Martin, for clockless, low-power, high-bandwidth switch chips. Fulcrum started shipping its PivotPoint ASICs in 2004, and had been pushing down latencies and pushing up bandwidth over the years. Fulcrum also created a switch that was programmable – something that Barefoot has made even greater strides with, and quite possibly because Intel didn’t do much with Fulcrum.
At the time Intel bought Fulcrum, the chip upstart had three different Ethernet ASICs. The “Alta” chips had 8 MB of buffer memory and delivered 72 ports running at 10 Gb/sec with 400 nanosecond port-to-port latency; they delivered 1.08 billion packets per second of aggregate throughput with L3 tunneling and significantly were the first ASICs used in 10 Gb/sec switches from upstart Arista Networks. The “Bali” chips cut that back to 24 ports running at 10 Gb/sec, cut the memory back to 2 MB, cut throughput for L3 routing back to 360 million packets per second, and dropped support for the Trill protocol and virtualization but dropped the latency down to 300 nanoseconds on a port hop. The “Tahoe” chips also delivered 24 ports running at 10 Gb/sec, but only delivered L2 forwarding and only had 1 MB of shared buffer memory, but the port-to-port hop was down at 200 nanoseconds. By comparison, InfiniBand latencies were in the range of 140 nanoseconds for 20 Gb/sec DDR speeds and 170 nanoseconds for 40 Gb/sec/56 GB/sec QDR speeds, and even today, with 100 Gb/sec EDR InfiniBand, latencies are only as low as 86 nanoseconds on that port hop. The point is this: Getting Ethernet down to 200 nanoseconds, even with a stripped down protocol stack, was an accomplishment.
The point is, those Fulcrum chips were interesting and we never heard from them again once Intel acquired the company. We suspect that won’t happen with Barefoot this time around. Perhaps Intel was distracted by its ambitions in the traditional HPC simulation and modeling market. In January 2012, only six months after buying Fulcrum, Intel spent $125 million to acquire the InfiniBand switch and adapter business from QLogic, which provides the foundation of its Omni-Path networking business, which is now delivering 100 Gb/sec ports and which is working to deliver 200 Gb/sec ports. The in April 2012, Intel bought the “Gemini” XT and “Aries” XC interconnect technologies from supercomputer maker Cray for $140 million.
The idea here was to merge QLogic InfiniBand and Aries into an uber-connect suitable for HPC and regular commercial customers, pushed by Cray and others. It is debatable that this will happen, particularly with Cray now going its own way with the “Slingshot” superset of Ethernet that it has created all by itself, using its own expertise in dynamic routing and congestion control layered onto a 64-port switch that runs at 200 Gb/sec. Cray doesn’t need Intel as much as Intel needs Cray, and it is no coincidence that two of the four exascale systems announced so far in the United States – “Aurora” at Argonne National Laboratory and “Frontier” at Oak Ridge National Laboratory are based on Slingshot, and a fifth pre-exascale system – “Perlmutter” at Lawrence Berkeley National Laboratory – is also using Slingshot. Supercomputer centers might want InfiniBand still, but these ones at least do not seem to want Omni-Path running at 200 Gb/sec, which Intel has promised to deliver around now.
Given all of this, it was no surprise to us that as 2019 was starting Intel was rumored to be interested in acquiring Mellanox, which would have given it a lock on the InfiniBand switching market as well as a very good Ethernet switching and adapter business – and one that we suspect makes more money from the hyperscalers and cloud builders than Barefoot does. And given that Mellanox is being acquired by Nvidia, which is a big rival in HPC and AI compute that will now have its own switching options for traditional HPC as well as hyperscaler and cloud builder customers, it is no surprise that Intel is reacting and buying Barefoot to get some more business at hyperscalers and cloud builders. And don’t be surprised to see Intel grab that P4 programmability of the Tofino switches to create highly tuned protocols that offload scatter-gather operations commonly used in HPC and AI workloads onto the switches – something that has been akin to heresy at Intel for the past decade and something that Barefoot has already been working on.
Which brings us all the way back to Barefoot, which we profiled in depth three years ago as it was just getting Tofino samples into the field and the related P4 programming language it had created in collaboration with other companies and academic institutions was starting to take off. Nick McKeown, chief scientist at Barefoot, is one of the company’s co-founders, and in addition to being a professor at Stanford University, where a lot of software defined technologies have originated, McKeown was also one of the founders of virtual networking upstart Nicira, which VMware acquired for $1.26 billion back in 2012 to get a virtual networking companion to its ESXi server virtualization and its VSAN storage virtualization. The foundation of Barefoot was laid back in 2008, when McKeown was collaborating with Martin Izzard and Pat Bosshart of Texas Instruments on programmable networking. When Barefoot was founded in 2011, McKeown became chief scientist, Izzard became chief executive officer, and Bosshart became chief technology officer. The original P4 paper published in 2013 concerning the programming of protocol-independent packet processors outlines the vision Barefoot has, taking the relatively simple abstraction of OpenFlow (which abstracts the forwarding tables from the switches) and moving it up a whole level higher to a full blown programming language that can control the data plane in switches and routers.
Barefoot announced its first generation of Tofino chips in the middle of 2016 and started shipping them in early 2017. The design had a block of 256 SERDES running at 25 Gb/sec plus four additional SERDES running at 25 Gb/sec to split downlinks from uplinks. That is an aggregate of 6.5 Tb/sec of switching capacity that can be carved up into chunks running at anywhere from 1 Gb/sec to 100 Gb/sec on the smaller block of SERDES or from 10 Gb/sec to 100 Gb/sec on the larger block in the top-end chip. Barefoot scaled the aggregate throughput at 1.9 Tb/sec, 2.5 Tb/sec, 3.3 Tb/sec, and 6.5 Tb/sec to address different brackets of bandwidth and port counts. Networking ODMs Accton (Edge-Core) and Wistron (WNC) started making whitebox switches based on Tofino at launch, and Arista Networks followed later in the summer. Cisco Systems inked a deal with Barefoot, too, for its Nexus 3400 switches but didn’t want to talk about it this time last year as its Cisco Live conference was underway, so we did it for them. Other switch makers, at the behest of cloud builders and hyperscalers, have probably built devices based on the Tofino 1 chips as well. Interestingly and predictably, the Tofino 1 chips were etched using Taiwan Semiconductor Manufacturing Corp’s 16 nanometer processes.
As 2018 was coming to a close, the Tofino 2 chips were launched, which not only scale up the bandwidth on the SERDES, which use PAM-4 encoding and 25 Gb/sec raw signaling to get to 50 Gb/sec effective per lane; by ganging up eight lanes, port speeds can go as high as 400 Gb/sec. (We have no idea what the latency is.) The Tofino 2 chips also scale the number of packet processing engines in the chip to deliver different performance on that vector. The Tofino chips come with 6.4 Tb/sec, 8 Tb/sec, and 12.8 Tb/sec of aggregate bandwidth and 24, 48, or 80 packet processing engines activated, as we detailed back in December 2018.
Interestingly, the Tofino 2 ASICs use a chiplet design, with the core die that does the packet processing etched using TSMC’s 7 nanometer processes with an “I/O out” configuration that puts the blocks of SERDES on the outside edges of the core and, we think, etched in a fatter 12 nanometer process from TSMC where the signaling is better. (It could be 16 nanometer, too.) This is the same approach AMD is taking with its “Rome” chips, which have cores etched in TSMC’s 7 nanometer technology but the memory controllers and I/O blocks etched using GlobalFoundries 14 nanometer processes. Barefoot can, by the way, snap in 50 Gb/sec native signaling plus PAM4 to boost the lane speed to 100 Gb/sec (after forward error correction, of course) when that becomes available. That would probably come with a Tofino2+ chip, not the Tofino 3.
And that brings up the first interesting thing about the Intel acquisition of Barefoot. Will Intel try to shift all of the manufacturing of Tofino 2 chips to its own 10 nanometer processes, or future 7 nanometer or 5 nanometer processes, or will it stay at the TSMC foundry. We highly doubt that Intel will mess with the Tofino 2, but it might do a Tofino 2+ and also move the wafer baking in-house at the same time, perhaps in late 2021 or early 2022, depending on how badly it wants to compete against datacenter switching juggernaut Broadcom.
Intel did not disclose what it paid to acquire Barefoot, but given the premium that Nvidia is paying for Mellanox (about 6X annual revenues) and the growing popularity of the P4 language for networking functions, we figure it has to be a pretty penny.
It is certainly enough to cover the money that has been pumped into Barefoot – and then some. Barefoot raised $1.35 million in seed money back in December 2013. Lightspeed Venture Partners and Sequoia Capital kicked in $24 million in Series A funds in May 2014. In June 2015, Dell and Andreessen Horowitz worked with these two investors to put another $50 million into the kitty, and in June 2016, when Barefoot dropped out of stealth and told people about the Tofino chips, Google and Goldman Sachs (which is big into Open Compute) plus the existing investors plus Hewlett Packard Enterprise and Hermes Growth Partners all ponied up another $57 million in Series C funding. That Series C round was extended to November 2016, when Chinese hyperscalers Alibaba and Tencent kicked in another $23 million to Barefoot. That is a haul of $155.4 million, which is not a lot in this day and age.
Any way you want to think about doing the math, Intel will spend billions of dollars on Barefoot, and it will eventually have to tell the US Securities and Exchange Commission because that amount, and the effect on Intel’s Data Center Group, is material. (So Intel could have just said the number and hoped like hell no one else came up with a competitive offer. If someone does want to make a competitive offer, they will whether or not Intel provides the number.)
Intel expects the Barefoot acquisition to close before the third quarter ends in September, which is moving pretty fast and which indicates that Intel is pretty sure what it paid is good enough to keep off competitors (as Nvidia did with Mellanox) and that the company is pretty sure the antitrust authorities are not going to balk.