The Lustre file system has been the canonical choice for the world’s largest supercomputers, but for the rest of high performance computing user base, it is moving beyond reach without the support and guidance it has had from its many backers, including most recently Intel, which dropped Lustre from its development ranks in mid-2017.
While Lustre users have seen the support story fall to pieces before, for many HPC shops, the need is greater than ever to look toward a fully supported scalable parallel file system that snaps well to easy to manage appliances. Some of these commercial HPC sites might have been holding the line for Intel to invest heavily and make Lustre more commercially attractive to meet reliability and other concerns, but as with many popular open source packages that stretch across markets, it is hard to build and support something that has dueling demands from users at the upper, middle, and lowest tiers. Instead of choosing one, Intel decided to drop the efforts entirely, sending Lustre out to pasture.
That is not to say the pasture is untended, but rather, each national lab is its own massive field, planted with lab-specific requirements. Cultivating Lustre at that scale and specificity does not generalize to a lot of back propagation to the larger community.
As open source approaches to storage flooded the general IT market in recent years, some expected that Lustre find new footing in large enterprise since it has been proven at scale for some of the world’s most complex deployments. While there are scattered examples of Lustre inside big enterprises, this vision never materialized, even with the commercialization boost Intel provided when it took the reins a few years ago to guide Lustre into its next frontier. Those commercialization and support plans have given way, however, and Lustre is once again finding its footing as an unowned and officially unsupported file system.
While the changing of the Lustre guard might not have an impact on continued use for the large national labs with the in-house development resources on hand to keep pushing file system, for smaller and mid-sized commercial high performance computing users, the need is greater than ever for a supported, simple to use approach to storage and I/O management. In other words, Lustre will be pushed by the very highest end of large-scale supercomputing but for that sizable HPC market with real business drivers that require stability, reliability, performance, and cost effectiveness, there is room for other entrants.
For those very few companies that fit the bill, including Panasas, which already has a strong commercial HPC business in shops that do not have the internal resources to roll their own complex Lustre deployments, this shift has been a long time coming.
“When Intel dropped Lustre it hardened the lines between that upper tier of users at the national labs and the rest of the HPC market,” says Faye Pairman, CEO of Panasas. The company has been actively engaged with HPC shops of all sizes since 2000 with its own proprietary parallel file system (PanFS) and ActiveStor scale-out storage appliance that have both hard drives and SSD capabilities. Most recently the company rolled out the latest incarnations of its ActiveStor Director, the ASD-100, and its ActiveStor Hybrid 100 (ASH-100), which we will go through in more detail in a future piece that covers some of the core differentiating technologies Panasas maintains.
It is natural to wonder why Panasas might take a shine to Lustre and snap it up to add to its HPC storage portfolio, but as Pairman explains, the reason for leaving it on the table is clear—it is just far too expensive to maintain and support a file system, particularly one that will be used at the top tier of computing for free by labs with their own development power in-house. She says that if anyone might snatch up Lustre and guide it to its next stage it would be one of the hardware makers eager to get a lock on its future, but whether it is a storage company like DDN or Cray, both with storage businesses and tight connections to the HPC systems market, the margins just are not there to support such a move—and the reward is not high enough. This means Lustre will persist at the top end, leaving the rest of the market free to explore what is possible with other file systems and packaged storage offerings.
“It is expensive to take on a file system. For a lot of companies in hardware, there’s little room for software development and that’s not just the talent needed to drive it forward but the capital equipment to test those systems and make them reliable.” Pairman tells The Next Platform. “It has to be kept modern, new features added—it takes a lot of engineers not just to support something like Lustre to forward it. That is a good part of the reason why Intel’s commercial efforts failed; those efforts were distinct from what the national labs were asking for. It is just too hard to meet all the community requirements and actually broaden it.”
Lustre’s hardening position at the high end of academic HPC versus the commercial space bodes well for Panasas, says Pairman. “In commercial HPC there is limited room for open source file systems. If you need HPC performance, commercial reliability and features, you will have to look to a commercial supplier. Open source software and software defined storage might work at the extremes—the low end or the high end—but for those with complex deployments that just need their storage to work and scale well, the answer is not open source.”
As for the hardware companies that are packaging Lustre to go down to the commercial market, there is another challenge. “When you need to have something that is a tight high performance system that is manageable and reliable, it means everything has to work together in an integrated way. That is difficult enough on its own, but taking into account the many purposes Lustre is trying to serve there is no ‘set it and forget it’ possible.”
With the position of Lustre in mind, we will explore that much broader HPC market Panasas is targeting and what HPC storage trends are on the horizon in the second of this special interview series.
Be the first to comment