Software wants to be free, or more precisely, the organizations that deploy software for specific functions in their platform stack want to be able to do so on the systems that they choose rather than the ones that a vendor that is fond of its hardware margins chooses. The hardware dependencies were broken in most servers long ago, and now the same disaggregation – and freedom – is coming to networks and storage.
The Isilon high-end storage unit of EMC, which does a fair amount of business in both the high performance computing and data analytics spaces that we track here at The Next Platform, has taken its first steps into the so-called “software-defined” world of hardware freedom.
EMC’s recent moves set the stage for deploying large-scale virtualized Isilon NAS arrays in the datacenter or on the public cloud, with the virtualization layer providing the common “hardware” substrate for the OneFS storage operating system at the heart of the Isilon storage clusters. EMC is starting out small, however, with a new Isilon cluster called SD Edge that runs on virtual machines and is intended for remote offices. The company has also rolled out a gateway for linking Isilon arrays to public cloud object storage, called Cloud Pools, but we think this is a precursor to EMC eventually offering virtualized Isilon arrays on various public clouds.
Server maker Dell, which has a pretty respectable storage and switch business and which is reinvigorating its high performance computing business, is in the process of buying EMC for $67 billion. Isilon, which is part of the storage company’s Emerging Technology division, is one of the crown jewels that makes EMC worth so much money. (We also think that ScaleIO hyperconverged storage brings significant value to Dell, although it will have to balance its own products against those from partners like Nutanix.) It is hard to say what will happen to Isilon in the wake of the Dell deal getting done, but it stands to reason that the Isilon and ScaleIO will play a more important role for the future of Dell than will more traditional SAN products like EMC’s VMAX arrays. The days of monolithic – and expensive – storage are numbered. Storage servers packed with cheap disk and a smattering of flash that can be equipped with different kinds of storage software as the needs arise is what companies will increasingly demand, and if they can get it open source with commercial support, some companies will be even happier.
We doubt very much that EMC will be open sourcing its storage software stack anytime soon, but it does look like EMC is tearing its software free of its own hardware, as the SD Edge software announced last month demonstrates.
“Think of this as the first step in an overall software-defined strategy for Isilon,” Sam Grocott, senior vice president of product management and marketing for EMC’s Emerging Technology division, tells The Next Platform. “This is 1.0 for Isilon, but absolutely we have heard our customers and this is the first of many products to come down the line. We believe in choice, and one of those choices has to be software-defined in addition to appliances. We really think the market is requiring that.”
This is a pretty big change for Isilon, which was founded in 2001 and got its first big business from the media and entertainment industry (much as rival DataDirect Networks did). These days, in addition to DDN, Isilon’s main rival is NetApp, which pioneered scale-out NAS, and of course the server giants that also sell NAS products (Hewlett-Packard Enterprise, Dell, IBM, and Oracle, pretty much in that order). In the HPC segment, as we have previously reported in the wake of the Dell-EMC deal, Dell is the leader in the server space and EMC is the leader in the storage space – something that would probably surprise a lot of people. (It did us, to be honest.) Needing a more competitive scale-out NAS offering, EMC bought Isilon five years ago for $2.25 billion, when the scale-out file and object storage business, for both appliances and for software-only setups, was around $2 billion a year. At the time of the deal, the combined EMC Atmos object storage and Isilon file and object storage were expected to have a run rate of $1 billion a year as 2012 came to a close. We have no idea if they made it, but considering that scale-out NAS was a $6.1 billion business in 2014 and that Isilon has 6,000 customers (a factor of 4X what it had five years ago), it is safe to say that Isilon is a growing and very likely very profitable business for EMC.
Shifting to software-only distributions would be dangerous for EMC to do alone, since it is not a server maker in its own right (except some dabbling here and there), but it fits perfectly with a combined Dell and EMC. Dell can treat Isilon like it does any other software stack at a hyperscaler, HPC center, or large enterprise: it is just software that needs to be co-designed with hardware to meet specific needs of individual customers.
The SD Edge software that Isilon has launched is intended for edge cases, literally, out there in branch and remote offices that need to share unstructured data with a central repository back in the corporate datacenter. The Isilon OneFS software has been virtualized and propped up on top of VMware’s ESXi hypervisor.
Architecturally, customers could in theory build a virtual Isilon NAS out of SD Edge as far as they want – EMC has placed no limits on its scale internally in the code – but in terms of its go-to-market strategy, EMC is aiming this at customers with under 36 TB of unstructured data. The machines supported have to be on the Virtual SAN hardware compatibility list as well, just so EMC doesn’t go nuts with a zillion different server, storage, and network options. Multiple servers running SD Edge are linked through Ethernet networks. The interesting bit is that the Isilon OneFS file systems and storage operating system runs in bare metal mode on the real Isilon appliances, and this is the first time it has been virtualized.
It is interesting to contemplate a completely virtualized and hardware agnostic Isilon platform. Something that will run on any X86 hardware and on any popular hypervisor and perhaps even on Docker containers, and perhaps on any public cloud, too. EMC did not announce this, and Grocott did not offer any suggestions that EMC would be heading in this direction, but it seems logical. Making Isilon storage clusters scale atop virtualized infrastructure may not be trivial.
And scale they do.
The Isilon platform is certified to scale up to 50 PB of capacity today in a single cluster with a single volume, but its architecture is able to scale further than that, says Grocott. (How far, he did not say. But 50 PB is a lot for everyone but the largest hyperscaler and HPC center.) The Isilon architecture scales compute for running OneFS and storage capacity as a unit within a server appliance, rather than hanging expansion enclosures off controllers as some storage architectures do, allowing for compute and storage to be scaled somewhat independently. The current upper limit for Isilon storage clusters is 144 nodes, and the nodes come in various sizes and capacities.
The Isilon X410 nodes have 36 3.5-inch drive bays and nodes are linked together using 40 Gb/sec QDR InfiniBand links. The X410 enclosure has two eight-core Xeon processors on its controller, with from 32 GB to 256 GB of server memory that is used as cache, and up to six of the drive bays can be equipped with SSD flash for storing hot data (particularly the metadata from the OneFS file system). The flash comes in capacities ranging from 400 GB to 1.6 TB. The SATA drives used to house colder data range in size from 1 TB to 4 TB and spin at the 7.2K RPM rates you expect from warm or relatively cold storage. (If you want to sacrifice some money and capacity for performance, the Isilon arrays support faster SAS drives.) The X410 controller also has 2 GB of non-volatile RAM for caching of the whitest hottest data as it streams in over the 10 Gb/sec Ethernet ports that link the Isilon storage cluster to the outside world; it can deliver 1.2 GB/sec of sustained throughput per node. With 144 of the X410s, Isilon NAS clusters top out at 20 PB of capacity. That 50 PB capacity upper limit is achieved through the Isilon HD400 enclosure, which crams 59 6 TB disk drives and one 800 GB flash drive into a 4U enclosure.
EMC tunes up the Isilon arrays for specific workloads. One is for file workloads with high transaction volumes and low latency, which are heavy on the CPU and flash. Another part of the line is focused on high concurrent throughput for data, which means giving up some latency and transaction volume for hundreds or thousands of users trying to access data at the same time on the cluster. This setup uses SATA and SSD drives. For deep archive, the Isilon arrays uses the fattest and cheapest SATA drives and no flash. Customers can configure these three different kinds of Isilon arrays into a single volume and management domain and then tier data across these different kinds of nodes. Customers can also run separate clusters with separate volumes and replicate data across them, too.
And, thanks to a new gateway function called Cloud Pools that is embedded with the latest OneFS release, they can also archive unstructured data out to Amazon Web Services, Microsoft Azure, and EMC’s own Virtustream under the covers, with the data still looking like it is locally stored on the Isilon cluster. (Presumably support for Google Cloud Platform is in the works.)
“The beautiful thing about OneFS is that it is in the same exact spot in the file system,” explains Grocott. “You don’t have to create a new mount point or network drive, it is always there and you are just going to have a different I/O experience.”
The Cloud Pools function will ship early next year; pricing is not available yet. While Cloud Pools is interesting and something that many customers will want, they may prefer to have a virtualized Isilon cluster running on compute infrastructure in a public cloud rather than a gateway. EMC has not promised to deliver this, but as we say, it seems logical. What we know is that companies are looking for ways to abstract their compute, storage, and networking in such a way that it can run on private infrastructure as well as public clouds with as little change to the stack as possible. This is an argument for containerizing OneFS and letting it run anywhere.
The Elephant In The Storage Room
The big play for Isilon in the past year has been to consolidate HPC workloads and data analytics workloads onto the same storage.
“Commercial HPC is what we focus on, and it has been a very strong use case for Isilon,” says Grocott. “Our largest market has been the media and entertainment segment, and that spans everything from content delivery to special effects rendering to large scale archive. All of the large post production houses and studios all use Isilon to play that role as their large scale file systems for their special effects and render farms.” High-throughput genomics sequencing is also often backed by Isilon arrays, Grocott adds, and oil and gas producers in the energy sector have Isilon clusters backing their reservoir modeling simulations. And now, Hadoop has gone native on Isilon, alongside the NFS and CIFS/SMB file and OpenStack Swift object protocols.
Supporting HDFS protocols on the Isilon arrays allows companies to scale Hadoop compute, which runs on its own cluster, separately from Hadoop storage, which runs on the Isilon machinery. The Isilon array looks like a local disk drive to the Hadoop cluster, and unlike the real HDFS, which has triple copies of data spread around a cluster, the Isilon arrays only need to store one copy because it uses Reed Solomon erasure coding to spread data across the nodes in the storage array. The OneFS storage operating system knows how to make use of Remote Direct Memory Access (RDMA) over the InfiniBand fabric to quickly pull data together and dispatch it from the nodes to the cluster. The upshot is that disk drives or a whole node can blow out and the data is still protected and recoverable in the cluster, although it takes some time to rebuild it.
The one thing that the Isilon cluster does not do is run the Hadoop framework and MapReduce compute method on the Xeon processors in its storage controllers. (There has been some experimentation running Hadoop on the same nodes that run parallel file systems like Lustre, GPFS, and Gluster.) Over time, this could also change. We can environ Dell-EMC creating a combined platform that mashes up compute and storage for different workloads, using Isilon for virtual NAS and ScaleIO for virtual SAN and taking a run at all of the workloads in the enterprise.
“….main rival is NetApp, which pioneered scale-out NAS”
dude…netapp still doesn’t have real scale-out NAS. They have what I call scale out management [up to 24 nodes]. It’s still ultimately traditional failover pairs of filers….not real scale out like most of us think of it.
and even that scale out management they didn’t even pioneer….they bought it from spinnaker. in 2004….sheesh….and only took 10 years to get not a real scale out platform….