Cloud Storage: More Than Instant Gratification and Near Infinite Scale

For more than a decade, the ease and elasticity of cloud storage has slowly been drawing enterprise users away from their beloved in-house datacenters. And that allure has become stronger as data volumes have grown and storage systems have become more complex.

That was the main thrust of WekaIO CTO Andy Watson, who offered his take on the on-premise/cloud storage dynamic as keynote at our Next I/O Platform event.

According to Watson, the long-term contraction in the storage segment is being offset by the uptake of storage in public clouds.  We should note that the contraction he’s referring to relates primarily to unit shipments. Storage capacity has continued to increase as the underlying media has gotten denser and denser.

Watson said there are valid reasons for staying on premise – and we’ll get to those in a moment – but according to him, cost is not one of them. “They aren’t called hyperscalers for nothing,” he explained. “They do have the benefit of the economy of scale.”

Plus, as Watson put it, “it’s a lot easier to just wave your credit card at a cloud provider than to build out your own infrastructure.” The myriad of decisions about how to design and outfit your storage (flash versus spinning disk, internal versus external, capacity versus IOPS, and so on) have conveniently been made by the cloud provider. As a result, the effort of planning and deploying storage, not to mention, the needed capital expenditure, can be completely avoided.

When Amazon launched its Simple Storage Servers (S3) in 2006, it came with service level guarantees that promised fourteen-nines (99.999999999999%) data integrity, a level that was nearly impossible to match on premise. As Watson pointed out, at the time, in-house storage was aiming for five-nines or perhaps six-nines, just for system uptime. To compete, storage vendors offered enterprise customers things like erasure coding and various disaster recovery schemes, but such features also increased system complexity.

Watson also noted that competition from Google, Microsoft, IBM and others drove Amazon to provide even better storage services. This took the form of more sophisticated tools and methods, such as automated tiering and advanced encryption, which again, were difficult to match on-premise in a cost-effective way.

Wanting to protect their own turf, in-house IT departments sought out guidance from the storage vendors, said Watson. According to him, they tended to react by offering questionable analysis that purported to show the cloud was more expensive compared to on-premise infrastructure. “I happened to be at one of those major storage companies at the time when things were really getting intense,” recalled Watson. “And I can assure you there was no shortage, internally, of denial and misinformation, which didn’t help.”

As a result, disenchanted and often misinformed system administrators argued against moving to the cloud. “The entrenched self-interest of the storage vendors, combined with the old guard IT department personnel, were on what I call the wrong side of history.”

Nonetheless, the ease of outsourcing storage continued to grow, often driven by individual developers who resorted to a “Shadow IT” model that circumvented their own IT departments. This growth helped create a feedback loop that had further ramifications, one of which was that as the cloud providers became bigger and richer, they were able to poach the best IT talent from the corporate world. That meant it became increasingly difficult for businesses to attract and retain this expertise in-house. And the fact that the cloud providers had bid up salaries meant these people were going to cost more, further burdening the on-premise model.

There are, however, valid reason for staying in-house, noted Watson, pointing to privacy concerns, and in Europe especially, sovereignty requirements. The privacy issue has been around since cloud computing began, and companies with especially sensitive data never really became comfortable with storing company secrets and other delicate data outside the confines of their local networks. And for those companies, storage providers began to offer the kinds of rapid-provisioning tools that mimicked cloud services.

To address privacy concerns, providers are responding by attempting to educate customers about cloud-native encryption and other types of protections. The European sovereignty issue is starting to be addressed in a number of ways, including the development of renewable energy infrastructure in these countries by Google (part of the sovereignty requirements) and the establishment of data trustees by Microsoft, although the company recently abandoned that model in Germany. All cloud providers, however, are pledging to comply with the EU’s General Data Protection Regulation (GDPR). This is still largely a work-in-progress, but the motivation for providers to meet their customers’ privacy and sovereignty requirements is enormous.

Another potential stumbling block for storing data in the cloud is the lack of extensive file system support.  A lot of this storage, like Amazon S3 is object based. Yes, clouds support the Hadoop Distributed File System (HDFS), but as Watson reminded everyone, that’s mainly for MapReduce workloads.

Amazon does have its own NFS-style Elastic File System (EFS), as well as FSx, a facility that provides support for Windows and Lustre files. Meanwhile Google announced it is acquiring Elastifile, a software company that provides file-based storage for the enterprise. Microsoft has yet to make any big moves with Azure, said Watson, but given what’s happening with the competition, he said they are worth watching.

In any case, he believes the cloud providers now realize the importance of file services for those enterprises that have thus far resisted off-premise storage. And again, there is plenty of motivation for the cloud companies to bring these potential customers into the fold.

“There is no reason not to expect this trend of moving to the cloud to continue to roll on, which may or may not be good news for storage vendors in the industry,” Watson concluded. “But I think from the customer perspective, it’s all good news.”