• Memory-Like Storage Means File Systems Must Change

    May 24, 2017

    The term software defined storage is in the new job title that Eric Barton has at DataDirect Networks, and he is a bit amused by this. As one of the creators of early parallel file systems for supercomputers and one of the people who took the Lustre file systems from a handful of supercomputing centers to one of the two main data management platforms for high performance computing, to a certain way of looking at it, Barton has always been doing software-defined storage.

    The world has just caught up with the idea.

    Now Barton, who is leaving Intel in the …

    Read more

  • Intel Shuts Down Lustre File System Business

    April 20, 2017

    Chip maker Intel is getting out of the business of trying to make money with a commercially supported release of the high-end Lustre parallel file system. Lustre is commonly used at HPC centers and is increasingly deployed by enterprises to take on their biggest file system jobs.

    But don’t jump too far to any other conclusions. The core development and support team, minus a few key people who have already left, remains at Intel and will be working on Lustre for the foreseeable future.

    Intel quietly announced its plans to shutter its Lustre commercialization efforts in a posting earlier this …

    Read more

  • One Small Shop, One Extreme HPC Storage Challenge

    February 7, 2017

    Being at the bleeding edge of computing in the life sciences does not always mean operating at extreme scale. For some shops, advancements in new data-generating scientific tools requires forward thinking at the infrastructure level—even if it doesn’t require a massive cluster with exotic architectures. We tend to cover much of what happens at the extreme scale of computing here, but it’s worth stepping back and observing how dramatic problems in HPC are addressed in much smaller environments.

    This “small shop, big problem” situation is familiar to the Van Andel Research Institute (VARI), which recently moved from a genomics and …

    Read more

  • Bolstering Lustre on ZFS: Highlights of Continuing Work

    January 11, 2017 Ken Strandberg

    The Zettabyte File System (ZFS), as a back-end file system to Lustre, has had support in Lustre for a long time. But in the last few years it has gained greater importance, likely due to Lustre’s push into enterprise and the increasing demands by both enterprise and non-enterprise IT to add more reliability and flexibility features to Lustre. So, ZFS has had significant traction in recent Lustre deployments.

    However, over the last 18 months, a few challenges have been the focus of several open source projects in the Lustre developer community to improve performance, align the enterprise-grade features in ZFS …

    Read more

  • DDN Turns The Crank On “Wolfcreek” Storage

    November 8, 2016

    In the high performance computing arena, the stress is always on performance. Anything and everything that can be done to try to make data retrieval and processing faster ultimately adds up to better simulations and models that more accurately reflect the reality we are trying to recreate and often cast forward in time to figure out what will happen next.

    Pushing performance is an interesting challenge here at the beginning of the 21st century, since a lot of server and storage components are commoditized and therefore available to others. The real engineering is coming up with innovative ways of putting …

    Read more

  • Lustre to DAOS: Machine Learning on Intel’s Platform

    May 23, 2016 Rob Farber

    Training a machine learning algorithm to accurately solve complex problems requires large amounts of data. The previous article discussed how scalable distributed parallel computing using a high-performance communications fabric like Intel Omni-Path Architecture (Intel OPA) is an essential part of what makes the training of deep learning on large complex datasets tractable in both the data center and within the cloud. Preparing large unstructured data sets for machine learning can be as intensive a task as the training process – especially for the file-system and storage subsystem(s). Starting (and restarting) big data training jobs using tens of thousands of clients …

    Read more

  • ARM, Open Source Feed Buzz Around HPC File System

    February 24, 2016

    There is but a small cadre of scalable parallel file systems and while the list might be small, weighing the relative benefits of each option against the available resources can be a challenge, as Sven Breuner and his HPC admin colleagues at the Fraunhofer Institute realized in 2004.

    They were certainly not the first or last organization to labor over the inevitable question of which of these scalable parallel file systems to choose, but they did take an interesting route. Notably, the limited choices have not changed over those years, either—nor have the list complaints about each.

    To be more …

    Read more

  • Improving HPC File System Performance and Data Integrity

    October 13, 2015 Doug Black

    The precise point of departure in the history of almost anything of significance – of a country, a company or a technology – is usually imperceptible. A new chapter in the continually evolving HPC market segment began about 10 years ago when a few HPC visionaries foresaw a shift – one that would run parallel with the everlasting drive for more compute power. It was the need for significantly improved high-performance data storage, protection, movement and bandwidth concomitant with processing speed.

    Of course, even in 2005 it was a truism of long-standing that “the world is drowning in data and …

    Read more

  • Panasas Expands Reach Beyond HPC Foundations

    August 14, 2015

    For any company in the high performance computing storage market, sixteen years is quite a run. Still private and still growing past its early years as a venture-backed, scale-out NAS company, Sunnyvale-based Panasas has managed to carve out a niche at the highest end of computing—and has turned an eye to the broader world of high performance commercial markets.

    With approximately 150 employees, Panasas is still led by the company’s founder and chief scientist, Garth Gibson, who raised the small company through the HPC storage ranks in the early 2000s to its stable position today. The basis for the Panasas …

    Read more

  • Bringing HPC and Hadoop Under the Same Cluster Umbrella

    July 7, 2015

    The computational capability of modern supercomputers, matched with the data handling abilities of the Hadoop framework, when done efficiently, creates a best of both worlds opportunity. This pairing is happening as traditional high performance computing (HPC) struggles with unprecedented volumes of data that are not always well suited for HPC architectures–and as the Hadoop ecosystem enters a golden age of maturity.

    There have been numerous attempts on the systems software side to create tools that make using both HPC and data-intensive applications work well on the same cluster, but it is still early days on that front. Hadoop and HPC, …

    Read more

Previous Articles

Mellanox

The Next Platform Weekly

Tap the stack to painlessly subscribe for a weekly email from The Next Platform, featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.