PCI-Express Flash Revamped To Compete With NVMe Drives

It was hyperscalers Facebook and Apple that put enterprise flash memory card maker Fusion-io on the map a few years ago as they used its storage to accelerate their massive databases, accounting for the majority of its revenues in 2011 and 2012 and a big chunk in 2013. That business from Facebook and Apple gave Fusion-io time to flesh out its products for the larger enterprise market. Last June, Fusion-io was snapped up by disk and flash storage maker SanDisk for $1.1 billion in an effort by the latter company to get more leverage in the broader enterprise flash market. Now SanDisk is bringing its own flash chips to bear on in a revamped ioMemory lineup and is being more aggressive about pricing to take on rivals Intel and Samsung.

With the updated ioMemory cards that SanDisk is launching, the company is switching to the flash memory chips that it makes through its foundry partnership with Toshiba, called Flash Forward, and specifically uses the 19 nanometer 1Y flash technology that has been in production since late 2013. (This process uses 2D flash memory layouts, not the more advanced and dense 3D layouts that are sampling now from the SanDisk-Toshiba partnership.) Brian Cox, senior director of enterprise storage solutions at SanDisk, tells The Next Platform that these 1Y flash chips are not just important to SanDisk customers because it allows the company to create lower-cost ioMemory flash cards for tucking into servers, but because it gives the company a predictable supply chain and cost structure for this flash and also provides better thermals per unit of capacity, too.

“The price drops are dramatic,” explains Cox. “On our smaller configurations we can drop the price by 61 percent, and on a larger configuration we are able to drop it 42 percent. The reason is that Fusion-io had gotten a bit of a bad rap for high prices, and we wanted to dispel that. People can justify that first card, but we want to make it so companies buy that next card now. This is PCI-Express flash for everybody.”

The prior generation of ioMemory flash cards from Fusion-io were based on 20 nanometer flash memory chips from Micron Technology, and to get the performance levels it needed while also using this memory in a dense configuration, Fusion-io hard to overclocks the PCI-Express slots such that they were burning 40 watts of juice. With its own 1Y flash memory, SanDisk is able to create a PCI-Express flash card – the ioMemory SX350 to give it a precise name – that does not have to be overclocked and that stays within the 25 watt power envelope of a normal PCI-Express x8 slot, says Cox.

“New technologies are always emerging, but datacenters are also extremely conservative. They are not going to roll out a new technology without testing it for three, six, or nine months first. If you are a Japanese bank, it will probably be a year or two, and a hyperscale company might do it sooner, but none of them are going to do it right away.”

Overclocking those PCI-Express required changes to the BIOS and system settings in servers, which makes it that much harder for broader adoption. While SanDisk and Fusion-io have over 7,000 customers who have bought a total of 250,000 ioMemory cards to date, the idea is to get ioMemory cards as a more standard offering and broaden their appeal.

Hewlett-Packard, Dell, IBM, Lenovo, Cisco Systems, and Fujitsu are key server OEM partners for the ioMemory cards, and EMC, NetApp, Dell, and HP are key server OEMs that embed them into their products. Obviously, some hyperscalers have also widely deployed ioMemory cards in the past, but they are also no doubt looking at NVM Express-style flash drives as well as the M.2 memory sticks as alternatives.

Microsoft has deployed the M.2 sticks in its Open Cloud Server and seems to prefer this to flash drives that hang off a controller or flash cards that plug into a slot. Others who adopt this Open Compute system will no doubt follow, particularly if 3D memory sticks in the M.2 form factor become available before too long and deliver 3 TB of capacity at an acceptable price and performance level. Cox tells The Next Platform that SanDisk sells M.2 memory on the consumer side of its business and is aware of what Microsoft is doing but is not going to talk about unannounced products. As for NVM Express, Cox says that “we are going to embrace NVMe, there is no doubt about that.” But for the moment, ioMemory cards can deliver better performance than NVMe flash drives as is.

For those who need very low latency on flash, SanDisk is selling 200 GB and 400 GB UlltraDIMM flash memory sticks, which were developed in conjunction with Diablo Technologies and which plug into regular DDR memory slots, that have write latencies on the order of 5 microseconds. Financial services companies like these for certain latency-sensitive applications. They have been certified on Huawei Technology, IBM/Lenovo, and Supermicro systems thus far and another one is about to be announced. But again, the UlltraDIMM cards need special tweaks in their BIOS so they can be seen as file storage devices even though they plug into main memory slots, while the new ioMemory SX350 can plug into any server that has a PCI-Express 3.0 x8 slot, and now doesn’t need any overclocking.

Beyond that, Cox says that SanDisk has taken some of the engineering from its Virtual Storage Layer, which allows a flash card to be addressed like memory instead of like a disk drive, and cooked up a non-volatile memory file system for Fusion-io cards. This software is in limited availability, with feedback coming from those early customers, and is expected to be generally available later this year.

“New technologies are always emerging, but datacenters are also extremely conservative,” explains Cox. “They are not going to roll out a new technology without testing it for three, six, or nine months first. If you are a Japanese bank, it will probably be a year or two, and a hyperscale company might do it sooner, but none of them are going to do it right away.”

The Feeds And Speeds

Generally speaking, says Cox, the new ioMemory SX350 flash cards have a little more capacity but twice the random read performance and twice the throughput of the ioDrive2 cards that they replace in the Fusion-io lineup at SanDisk. So this is not just merely a matter of changing out the flash memory and dropping the price. That’s up to 350,000 I/O operations per second (IOPS) on random reads and 385,000 IOPS on random writes (on the 3.2 TB model) with a 15 nanosecond write latency and 2.8 GB/sec of read bandwidth.

There are four variants of the ioMemory SX350, ranging in size between 1.25 TB and 6.4 TB of capacity:

SanDisk does not provide list pricing for Fusion-io cards (and Fusion-io never did, either), but if you work the price/performance figures backwards that SanDisk is providing, then the ioMemory SX350 with 1.3 TB of capacity has a price of around $5,800 and the fatter 3.2 TB flash card has a list price of around $14,150. Buying three of the small cards is not a better deal than buying one of the fat ones, but depending on the physical configuration of a system, some customers might have to do it that way. (The fatter card is a lot taller than the skinny one.) SanDisk is also partnering with HP and Cisco to offer mezzanine card versions of the SX350 flash cards that plug directly into their respective BladeSystem and Unified Computing System machines.

To help make the case for the ioMemory SX350 drives, Cox brought out a slew of benchmark test results that pit the new flash card against NVMe drives from Intel and Samsung. The chart below shows the relative performance, in terms of I/O operations per second (IOPS), on a 512 byte benchmark test that Cox says is reflective of server virtualization environments that have a lot of small I/O. On the right is a throughput test using larger 128 KB file sizes and measuring aggregate bandwidth for the three devices.

This one shows the write latency of the three devices as the block side changes:

And this one shows read latency over the same array of block sizes:

While the benchmark tests across products are helpful in a very loose sense, The Next Platform thinks that customers should always run their own bakeoffs to see what the actual performance boost for their own workloads are for any particular technology. SanDisk can’t provide customer benchmarks without their approval, and enterprise customers are understandably cagey about giving out any data about their applications for competitive reasons.

But SanDisk has polled customers using Fusion-io flash memory cards to ask them how much acceleration they saw when adding flash to their disk-based systems or moving entirely to flash, we presume. Anyway, across those who were polled, 14 percent of customers said they saw a 2X boost, 32 percent said they got a 3X to 5X boost, 23 percent said they had a 6X to 9X boost, and 29 percent said they had a 10X or greater boost.