Impatient For Fabrics, Micron Forges Its Own NVM-Express Arrays
May 8, 2017 Timothy Prickett Morgan
There may be a shortage in the supply of DRAM main memory and NAND flash memory that is having an adverse effect on the server and storage markets, but there is no shortage of vendors who are trying to push the envelope on clustered storage using a mix of these memories and others such as the impending 3D XPoint.
Micron Technology, which makes and sells all three of these types of memories, is so impatient with the rate of technological advancement in clustered flash arrays based on the NVM-Express protocol that it decided to engineer and launch its own product line.
This is something that you don’t see a component supplier do very often, but it does happen. Intel has sold servers on and off for years under special circumstances to specific customers, and Nvidia peddles its hybrid CPU-GPU DGX-1 systems, to name just two examples. It is always a bit peculiar when such vendors do this, and it is often because they want to help set the pace and speed up access to a technology so much that they just do it themselves.
So it is with the SolidScale all-flash arrays from Micron, which were recently unveiled and which it hopes to get into the hands of early adopters later this fall and make generally available to any and all takers in early 2018. That is a pretty long lead time to tell customers about a product, too, and it is an indicator of just how eager Micron is to see its NVM-Express flash and QuantX 3D XPoint storage, which the SolidScale system is designed from the get-go to support, business take off.
The all-flash array market is a crowded field with some pretty strong upstarts and incumbents, so it is hard to say how Micron will do as an array supplier. But no matter what, the SolidScale arrays provide an object lesson in what is wrong with non-volatile storage today as it is commonly deployed and what it might take to fix the problems. Andy Fisher, the senior product line manager at Micron who joined the company a year and a half ago after stints at Compaq and Hewlett Packard Enterprise, walked The Next Platform through the issues and the techniques that Micron has come up with the resolve them.
Given that Micron sells flash and soon 3D XPoint storage devices as well as flash memory to those who also want to sell such devices in turn to datacenters, you might think Micron would just keep mum about the whole issue of the inefficient use of non-volatile storage in servers these days. Micron wants to fix these issues to drive adoption, just like Intel set its techies to work on the NVM-Express protocol several years ago to get the heritage SAS and SCSI protocol stack out of the link between the central processor and the flash memory, which is not a rotating media. NVM-Express chops out all that legacy junk and allows for flash devices to more directly link to the compute complex over the PCI-Express peripheral bus, which increases the effective bandwidth and reduces the latency of I/O operations between the CPU and the flash.
Right now, there are a number of issues that are keeping NVM-Express storage from being widely adopted, and Fisher rattled them off for us. Servers in the “Haswell” and “Broadwell” Xeon generations – meaning over the past three years – might have had two or four NVM-Express ports hanging off the processing complex. This was good for a higher speed caching layer within the system, but it was not ideal in that all flash storage could not link in this fashion. Other issues that NVM-Express faces are the relatively high cost between regular SSDs and those with NVM-Express ports (something Micron could do something about in the future but has made no promises in this regard), but this premium is somewhat mitigated by the fact that you do not need an intermediate storage controller between the flash and the CPU on the peripheral bus. The relative immaturity of the NVM-Express over Fabrics extension to the protocol is an issue, but the plan is to allow for a pool of non-volatile storage to be accessible to multiple servers using Ethernet or InfiniBand networks that link the storage and the servers together in a mesh.
The other issues that NVM-Express flash arrays face are more architectural. “Datacenter CPUs are being underutilized, and we believe the cores are idle because they are starved for data,” Fisher explains. “More specifically with NVM-Express SSDs, they are screaming fast with microsecond latency, and they are really exciting if you put them into a server and tie them to a specific application, such as database or analytics or HPC, but the problem is that this direct attached storage model makes it hard to move the data and the applications, and even worse, you tend to strange capacity and IOPS. With NVM-Express SSDs, they are so much faster than hard drives – there are very few applications that can consume the IOPS when each drive in the system can deliver 750,000 IOPS. If NVM-Express is going to go mainstream, we have to get them out of an unshared, stranded architecture.”
That, in a nutshell, is what the SolidScale arrays are all about, and the idea is to have multiple enclosures of flash that can be clustered together for both capacity and performance scale out and that can be accessed at essentially the same latency as locally attached SSDs inside of the server.
Micron is not revealing all of the details on the SolidScale all-flash arrays just yet, but the initial machines are based on a two-socket Broadwell Xeon server; Micron is not divulging who its server hardware partner is, but like many companies, it is going to a third party. Each processor in the SolidScale controller manages a dozen NVM-Express drives, although in theory, Micron can scale this up to 128 devices and petabytes of scale for HPC applications if necessary. Each node has a pair of 100 Gb/sec Ethernet ports, and Micron has tapped Mellanox Technologies for these adapters. While the enclosure has 24 NVM-Express drive bays, it only takes four of its 9100 series NVM-Express SSDs to saturate these network ports. In any event, with a three-node base configuration, there is 600 Gb/sec of aggregate network bandwidth coming out of the SolidScale array cluster, and there is plenty of bandwidth to spare to hook out to the servers as well. With those three nodes, the array cluster can deliver 10.9 million IOPS in aggregate, and because of the software stack that is running on the arrays and the extended NVM-Express protocol that Micron is using, the incremental latency of the pooled storage compared to using the same 9100 series SSDs locally in the server nodes is 1 percent or less. This is, in effect, not difference.
Here is what the performance looks like with a real workload, in this case the Yahoo Cloud Serving Benchmark (YCSB) developed by the search engine company of that name. The tests that Micron ran used NVM-Express SSDs both locally and remotely using the SolidScale arrays, with the Apache Cassandra NoSQL database developed by Facebook:
Moving the flash out of the servers and into the SolidScale dropped the IOPS on the YCSB tests by between 4 percent and 8 percent, depending on the mix of reads and writes chosen, as you can see. Fisher also told us, as another data point, that in a test storing the databases for Microsoft’s SQL Server running atop a Linux system, a three-node SolidScale array was able to deliver 11.1 GB/sec of bandwidth into and out of the database.
The SolidScale architecture is designed to scale to over 1,000 nodes, which would yield 2.5 PB of aggregate capacity using its top-end 3.2 TB U.2 flash SSDs. By this time next year, when the SolidScale arrays are shipping, odds are that Micron will be able to at least double that capacity.
Here is the interesting bit. With the NVM-Express over Fabrics protocol only at the 1.0 release level and not really ready for primetime yet, Micron had to do something to allow high bandwidth, low latency access to a shared pool of flash storage. And that something was to partner with software-defined storage upstart Excelero, whom we told you about recently, to bundle its NVMesh software on the SolidScale iron. Fisher says that the two have worked to tune NVMesh for the Micron flash, and that the two have worked together on bits and pieces of the stack to make it hum. The other interesting bit is that Micron is willing to partner with other server and storage makers to resell this software atop their iron, and that the two are working closely to get it all ready for QuantX 3D XPoint memory.
What Micron has not done as yet is decide what price to charge for SolidScale, but what we do know is that it will have to compete against a lot of different players who all want to ride the NVM-Express wave into the datacenter.