File system changes in high performance computing, take time. Good file systems are long-lived. It took several years for some parallel file systems to win out over others and it will be many more decades before the file system as we know it is replaced by something entirely different.
In the meantime, however, there are important points to consider for real-world production HPC deployments that go beyond mere performance comparisons, especially as these workloads grow more complex, become more common, and put new pressures on storage and I/O systems.
The right combination of performance, stability, reliability, and ease of management matter a great deal when trying to obtain the required results from the most critical, competitive data stored by an organization. This is especially important for companies who don’t have the time or talent pool to manage the growing complexities of today’s workloads.
As we discussed in a recent article, the concerns of a typical HPC customer are often far different from research-only supercomputing sites. While an open source file system like Lustre delivers flexibility and scalability and might be a good fit for the national labs and their legions of technical support staff, most on-the-ground HPC data centers require a storage system that simply works—and does so without compromise to the performance requirements that separate HPC from the rest of the enterprise workload world.
As Panasas CEO, Faye Pairman, tells The Next Platform, there is immense value in the work done on storage and I/O systems at the top 500 supercomputing sites, but the next-generation technologies that are envisioned and being developed are completely off the horizon for most HPC customers. While it is interesting to consider what comes after file systems and what will work for the next decade’s supercomputers, the file system as we know it is here to stay for the long haul for the bulk of HPC workloads, whether in a traditional or in an Enterprise environment.
With all of this in mind, it is clear there is a sustainable growth potential for a storage approach and supported parallel file system that can focus on this critical set of users in HPC.
“The lines between traditional HPC and Enterprise applications are blurring,” Pairman explains. “New technologies such as machine learning, deep learning, AI and virtual reality are supporting the growth and expansion of HPC workloads , and organizations adopting these technologies are now finding themselves dealing with HPC type requirements in their data centers. What these organizations need are agile, multi-petabyte deployments that are up and running in hours and can be managed by small IT teams.”
Panasas has held its own ground in commercial high performance markets since around 2000, with big name customers like Boeing, Honeywell, NASA, Mercedes-Benz, Lockheed Martin, Airbus, P&G, and many others across industry and research. Most recently, the company’s own parallel file system, PanFS and associated ActiveStor appliances have served as the backbone for managing large files and scalable throughput performance with increasing focus on mixed workloads—a challenging subset of HPC that is more common and requires manageable complexity from storage and I/O systems.
The customers Panasas focuses on, including those with names that do not grace the Fortune 100, look for a fully supported solution and usually cannot find the performance or support they need from software-only solutions, moving work entirely to the cloud, or through complex open source file systems, no matter the cost benefits those alternatives might offer.
“The requirements of our customer segments are specific and highly technical, which means that the space we play in is limited to players who can meet some of the most challenging market and product requirements,” Pairman adds.
While some of the companies Panasas works with do have systems that sit at the bleeding edge of supercomputing, for many of their daily operations the real driver is having the right combination of performance, reliability, and management ease. In terms of trends at midsized and smaller HPC shops (or smaller systems within larger companies) are shifts that are happening elsewhere in the enterprise datacenter. While there is some adoption of InfiniBand and high-end networks, Ethernet still reigns. Adoption of new processors brings no unexpected surprises, and while flash is interesting to many companies, the price point is still prohibitive for predominately large file deployments. There is opportunity for an HPC oriented company to address the needs of the Enterprise and Panasas is determined to make its mark.
“What our customers want is for their storage to be invisible,” Pairman explains. “They want no performance bottlenecks, capable 24/7 support , and they want their systems to be up and running at all times. We are unique in that we make the complex simple for the user.”
This is why, as we noted in the first article in this series, that Lustre has not developed legs in the enterprise, and is why companies like Panasas will continue to have a big role to play as HPC continues to go commercial.
Unlike general enterprise, the technologies required to meet mixed and complex workload demands require a sophisticated approach to both hardware and software. In the next article in this series, we will go beyond industry trends and what commercial HPC users require from storage and I/O systems and drill into the unique technologies Panasas has put in front of the market for 2018.