There are so many barriers to capacity increases in high performance computing systems that it would take a long time to rattle them off. But one of the most important ones, which we have covered in-depth with many experts in recent years, is the day that electrical signaling can no longer be used to get data into and out of a compute engine or network ASIC.
With each increase in bandwidth for these devices, there is a consequent shortening of the distance that an electrical signal can be pushed. Thus far, materials scientists and PHY and SerDes designers have been able to keep that day at bay, but sooner or later, the applicable sections of the Periodic Table just run out of atoms to do tricks with, and that is why the development of silicon photonics is so important.
Hewlett Packard Enterprise is no slouch when it comes to optical communications, which among other things were at the heart of The Machine, a memory-centric parallel system that was built by HP Labs many years ago as a test bed for optical interconnects and memory pooling. HPE was also behind the Gen-Z protocol and even created an optical Gen-Z switch and electro-optical Gen-Z adapters for servers, which we detailed back in September 2019, under the auspices of the US Department of Energy’s PathForward program. Soon thereafter, Dell, which was one of the big proponents of Gen-Z along with rival HPE, showed off a memory server based on Gen-Z technology.
But when it considered how it might layer on silicon photonics to future generations of the switch ASICs and network interfaces of its Slingshot Ethernet interconnect, which we talked about recently with HPE, the system maker decided to partner with Ayar Labs, a silicon photonics startup that was funded by the US Defense Advanced Research Projects Agency and that taped out its first electro-optical chips in early 2019 and that we learned a lot about at the Hot Chips 2019 conference later that summer. Ayar Labs believes that beyond 112 Gb/sec native signaling, which translates into 100 Gb/sec after encoding overhead is taken off, electrical signaling will hit a wall. This sentiment was espoused by Andy Bechtolsheim (of Arista Networks, Granite Systems, and Sun Microsystems fame) four years ago, as we pointed out in this story originally, but now it looks like sometime in 2025 electrical signaling will get yet another reprieve and move ahead to 224 Gb/sec (200 Gb/sec after encoding) native. The barrier has moved. But at some point, and in the next few years, it will be a barrier for real.
Each switch ASIC and each network adapter will make use of the combination of 50 Gb/sec, 100 Gb/sec, and the future 200 Gb/sec signaling, NRZ and PAM-4 encoding (and perhaps other levels of pulse amplitude modulation encoding), and lane counts to create 200 Gb/sec, 400 Gb/sec, 800 Gb/sec, and maybe even 1.6 Tb/sec devices. But based on research and prototypes that have already been created by Intel’s Barefoot Networks networking team and Broadcom’s silicon photonics team, we think that this electrical signaling wall will have to be breached by silicon photonics in the next one or two generations of interconnects. (We mean on the roadmap, not the ones coming out of the fabs soon.)
That’s not far away, and hence the partnership between HPE and Ayar Labs.
“Ultimately, this is not a unique problem to Slingshot,” Marten Terpstra, senior director of product management for the High Performance Networks, HPC, and AI business group at HPE, tells The Next Platform. “We know that somewhere in one of these next generations – and I am not going to be explicit about which one – we are going to run into limitations of bandwidth across an electrical interconnect, which affects ASICs, whether they are a CPU, a GPU, a switch chip. The faster you go, the more dense you make the device, the issues with power, density, complexity, and signal integrity are going to be so significant that electrical signaling isn’t a commercially viable solution anymore. The tipping point is coming that somewhere in the next few generations – we are not being very explicit yet as to what generation, but think about it as being somewhere in the next few generations – we will hit this point.”
Having never moved The Machine beyond research and prototype and having rolled all of the Gen-Z intellectual property into the CXL Consortium mere months ago, HPE is in no mood to try to go this jump to silicon photonics alone.
“If you look at what 400 Gb/sec or 800 Gb/sec switches look like, the internal design is getting more complicated,” explains Terpstra. “In the past, with switch designs at lower bandwidths, you had a switch ASIC and you could simply run the electrical signals all the way to the front panel. In some cases, you can’t do that anymore, or you have to run special cables. We are going to run into that issue with the Slingshot portfolio, and we have been talking to Ayar Labs for a little more than a year, and they are leading the charge in this domain by creating optical interconnects based on silicon photonics and integrated packaging. And they are not just going to create components, they actually want to create an ecosystem, and that makes them a perfect partner for us. Other component suppliers will need the same kind of integrated, co-packaged silicon photonics. We have to have heterogeneous solutions, and we cannot come up with something that is proprietary. There’s some resistance to that kind of methodology. We have ten years of silicon photonics research and we could have productized that, but we deliberately chose not to do that. What we really need is an industry alignment that is broad so we can actually create legitimate heterogeneous systems.”
That, says Hugo Saleh, senior vice president of commercial operations and managing director of Ayar Labs UK, is exactly what Ayar Labs wants to do.
“Since we transmit light out of the package, what we want to focus on is building an ecosystem that can align on how many wavelengths are used, how far can the wavelengths be from each other, and how tall are the peaks,” explains Saleh. “We are excited about our optical I/O and the chiplets that we build, but we are focusing on the characteristics of the light so that we can allow for the interoperability of vendors down the road. To make this ecosystem flourish, there have to be other companies in this business. That is why we are so excited to be working with HPE because they can set expectations and put requirements on suppliers.”
Ayar Labs has been working with Intel in some capacity since dropping out of stealth mode, and it is not unreasonable to expect that a few hyperscalers and cloud builders might be kicking the tires and that Dell, Inspur, Lenovo, and IBM could follow suit. Cisco Systems might be a little more difficult because it has its own networking business. But as we have discussed a number of times, silicon photonics is going to be one of the key technologies in bringing disaggregated and composable infrastructure that can span an entire datacenter.
As part of the partnership deal with Ayar Labs, Hewlett Packard Pathfinder, the venture capital arm of the system maker, is kicking in an undisclosed amount of funding into the SiPho upstart. Ayar Labs has raised $64.5 million in seed money as well as two rounds of venture funding (not including the investment by HPE), and has Intel Capital and Globalfoundries as investors alongside BlueSky Capital, Applied Ventures, Castor Ventures, and Lockheed Martin Ventures, among others.