A New Architecture For NVM-Express

NVM-Express is the latest hot thing in storage, with server and storage array vendors big and small making a mad dash to bring the protocol into their products and get an advantage in what promises to be a fast-growing market.

With the rapid rise in the amount of data being generated and processed, and the growth of such technologies as artificial intelligence and machine learning in managing and processing the data, demand for faster speeds and lower latency in flash and other non-volatile memory will continue to increase in the coming years, and established companies like Dell EMC, NetApp, IBM and Intel and relatively new vendors like Pure Storage and Excelero want to make sure they are at the tip of the spear as the market expands.

E8 Storage is another one of those newer vendors that is working to find traction in the NVM-Express space. The company was founded in December 2014 by Zivan Ori, the company’s chief executive officer, and Alex Friedman, its senior vice president of research and development. Both are veterans of the IBM XIV storage business, as are other executives, including vice president of architecture Ziv Serlin and R&D director Danny Melamed. Their resumes also include such companies as Intel, Stratoscale, and Wanova, and other key employees come from storage vendors such as NetApp, BlueArc, and Nexenta.

The E8 founders started out with the idea that to best take advantage of NVM-Express, what was needed was a new architecture designed to leverage the capabilities and promises of the protocol, the first specification of which was released almost seven years ago.

“Most of the flash architectures out there are still built around the dual controllers, and those controllers are the bottleneck,” Julie Herd, director of technical marketing at E8, told The Next Platform. “No matter what happens, all the data, all the control goes through those controllers, so you are not seeing those controllers get the full performance out of the SSDs we have today. Adding NVM-Express doesn’t solve that. It just means you’ve got more SSDs that you’re not getting the full performance from. That’s why they knew they really wanted to get rid of that bottleneck in the dual controller architecture. That’s why we took a different approach to the architecture. It was designed for high-performance NVM-Express.”

Despite the deep experiences of the company’s founders with XIV, which was an independent company before IBM bought it for an undisclosed amount back in January 2008, “nothing is as far as the architecture comes from IBM XIV. We’re actually taking a very different approach to what is really a traditional flash array,” Herd says.

The company has spent the past three years building out its software platform that includes E8 agents that run on a host, offloading almost 90 percent of the data path operations from the controllers and up onto the agent. The metadata server, management, web-base GUI and high availability remains on and is managed by the controllers. However, the data path is offloaded onto the E8 host agent, which Herd says helps drive the scalability the vendor can offer. The platform can scale up to 96 host agents per E8 appliance, and up to four appliances per E8 instance. The E8 host agents can auto-discover assigned LUNs and operate independently from the dual controllers, which come with RAID 6 protection.

The key benefit to NVM-Express ports, beyond being faster than SATA and SAS ports, is parallelism, which Herd says the company was able to take advantage of. The E8 storage supports up to 96 hosts, and the NVM-Express drives can have up to 128 connections, instead of one drive that you have with SATA, and up to 64,000 queues with 64,000 commands per queue.

“That is exponentially more throughput capabilities in the protocol than you have with SAS or SATA,” says Herd. “We had to unlock the performance of all of those lanes, all of those queues all of those commands. That’s why we wanted to separate the control from those data paths, so that each of those hosts could have direct access using RDMA to the drive. They could do a direct write or a direct read to get all of those command paths running in parallel. You couldn’t do that if everything was going through those controllers from the data path. Now each E8 agent has the ability to, using RDMA Over Converged Ethernet to have a direct connection to the drive inside the controllers. By using network interface cards with RDMA enabled, we are able to get the hardware acceleration of those NICs as well on the host side as well as our controllers. By having the agents on each of the hosts, we have expanded our performance. Each host has its own processing, has its own throughput into the overall architectures. We’ve added reliability because of those hosts is responsible for its own data – it doesn’t affect the data for any of the other hosts, so if the host loses connectivity, all of the other hosts remain available, and it doesn’t take down the controller either.”

E8’s storage software runs atop appliances built by AIC, a Taiwan-based original design manufacturer, using Intel Xeon processors, and its first system, the S10, is primarily used for proof-of-concepts. But in August 2016, E8 launched the D24, a rack-scale flash appliance that uses NVM-Express drives that the company said at the time provided 10X the performance of other flash arrays, and was less expensive due to E8 using off-the-shelf NVM-Express drives and Ethernet NICs. The system is used in most production environments, Herd says. A year later, E8 rolled out the E8-X24 appliance that uses Intel’s dual-port Optane SSDs.

E8 is expecting the adoption of NVM-Express to accelerate in the coming months, particularly given the expansion of the underlying hardware and the expected drop in prices.

“We’re seeing more vendors from the ODM space, more vendors in the channel market space, expansion in the NVM-Express drive options,” she says. “When we went to market, we went to market with Intel because they were the only dual-port-based NVM-Express we could find. Over the past year that has expanded. All of our partners have dual-port options now. Not only have the options expanded, now that we’re past the NAND shortages, we’re now going to see those NVM-Express price curves converged. It already started to happen before this shortage with the single port, now we’re doing to start seeing it happen with the dual port, especially as the larger vendors and the established players also start bringing out their own NVM-Express options. They will be the ones that will help bring those dual-port prices down. Because NVM-Express is a much more streamlined protocol, it has less overhead, it has better latency, better throughput, better performance, and if we have it at the same cost, you’re going to see everyone shift over to NVM-Express. It’s not going to happen overnight. Any type of protocol transition takes years.”

E8 is targeting a niche market that includes companies for whom low latency is key, Herd says. That includes customers relying on real-time analytics, such as financial services firms and companies do day trading that rely on real-time statistics to make their decisions. E8 has a number of Wall Street customers that are a year into production with E8’s technology. In addition, service providers, online hosting companies, media and entertainment and online travel sites are other examples of industries that are showing interest in E8. One online travel site is using the technology for real-time bookings; another for scanning of airline tickets.

The company also is working with IBM with its Spectrum Scale, formerly known as General Parallel File System, so that it has a solution for file-based workloads. “E8 Storage by itself is a block system. GPFS, though, is a very scalable, high-throughput, low-latency offering, and it’s going to allow us to tackle things like entertainment, real-time 4K video editing live, on the system,” Herd says.

 

By using IBM’s Spectrum Scale 5.0 nodes, E8 was able to beat other vendors like NetApp in the SPEC SFS2014_swbuild benchmark, include achieving 2.5 times more builds than an IBM all-flash array, eight times lower latency than the previous record and 0.69 ORT (overall response time). E8 released the benchmark results this week, which are shown below:

E8 says it will keep a laser focus on its niche market.

“If they’re putting us up in a competition against Dell, we’re talking to the wrong customer because if a Dell server right now will do what they need to do – or a Dell storage appliance – then performance isn’t their pain point,” Herd explains, noting that hedge funds that want performance at all cost is where E8 is concentrating its efforts. “If flash storage is good enough, then the customer is not for us.”

Many companies have strong relationships with their established vendors and will always lean in that direction. But while E8’s serviceable market is small, it will grow, she said. Proof of concept activity for the vendor already is increasing since the turn of the new year, indicating growing interest in what it offers. E8 wants to dominate that niche market, because in a world that has a lot of competitors – including large top-tier vendors – companies like E8 need to stay focused, she says.