AMD, Cray, Nvidia Behind Massive NCAR Supercomputer Upgrade

The National Center for Atmospheric Research (NCAR) is getting a significant upgrade to its all-CPU “Cheyenne” supercomputer, bringing it from 5.2 peak petaflops of performance to 19.87 petaflops with a combination of AMD CPUs and Nvidia A100 GPUs laid out in a Cray XE (formerly called “Shasta”) system from Hewlett Packard Enterprise.

The upgraded system will cost around $35 million and will complement the original SGI 8600-based machines that comprised the original “Cheyenne” system. NCAR’s director of HPC, Irfan Elahi, tells us that the competitive bidding process included IBM, Dell, Cray (which had not yet been acquired and had not brought the “Shasta” architecture to the table) and HPE with HPE winning on what it could provide based on the center’s requirements for 3X the performance of the original machine. In fact, based on the final specs for the machine, NCAR will be looking at closer to 3.5X the performance.

We asked Elahi about the decision to make the jump to all-Intel CPUs to AMD “Milan” with high-end A100 GPUs. “One of the main reasons we have AMD coming to this system is because the ‘Milan’ CPU and form factor would be ready on the Cray XE system in our timeframe. Second, the price performance for the AMD, especially from a Cray systems perspective, would easily meet our 3X target.” He adds that HPE did a lot of the benchmarking of NCAR codes and were the ones who came to the conclusion that it would be the best architectural mix for their weather and climate modeling workloads.

In terms of the GPU distribution on the new system (currently without an final name) NCAR’s requirements were a bit different. The RFP stated that 80% of the performance capability should come from CPUs with the remainder for GPUs. Of the 2570 nodes, there are only 82 with A100s (4X per node) but this satisfies the 20% requirement. Elahi says that as a result of the GPUs and other energy-efficient features, the new NWSC system will use just  40 percent more electricity than Cheyenne – which is itself highly energy efficient – despite  being almost 3.5 times faster.

Elahi says that major codes like MPAS-A have been ported to GPUs using the OpenACC programming model championed by Oak Ridge National Laboratory, Cray, Nvidia, and others. And in fact, MPAS is the engine that underpins the  GRAF model created by IBM and its The Weather Channel division. Still, broad GPU adoption in weather is still some time away—definitely after the machine arrives. He adds that having robust GPUs will let them begin to think about integrating AI/ML into some of their weather and climate workflows. All of these trends for GPUs in weather are happening around the world now although GPU acceleration is still not mainstream in this area of HPC.

Elahi says that many of the codes at NCAR are deeply memory intensive. The overall system has 692 TB of total memory. This is also the first time the team will have access to data movement enhancements including NVLink for the GPU work and of course, the Slingshot interconnect, something Elahi thinks will make a remarkable difference. “One of the things about Slingshot is that it is more open, so we are expecting we will be able to connect our visualization and data analysis resources to this cluster, that way we’ll have more seamless data movement between the analysis and the compute. We’re excited but there will always be challenges; Slingshot is new, we’re going in with our eyes open and we’re optimistic this will become the interconnect for Cray going forward. It’s challenging, it’s new, but it’s interesting.”

Cray had been the leader in systems sales to weather forecasting centers and in climate modeling in the last several years. Seeing NCAR join the ranks of what is now HPE Cray is not surprising, nor is the speculation about how difficult the learning curve will be for NCAR and how much of a difference it will make over the standard SGI server/network world NCAR has lived in over the last several years.

Storage is also an HPE Cray ClusterStor E1000 system with 60 PB of usable file system, which can be expanded through various management options up to 120 PB. The new system will have 5,088 16TB drives, a 40TB SSD for Lustre metadata and two metadata management units. There will be 300 GB/s aggregate I/O bandwidth to and from the system. Elahi says with the original Cheyenne deployment they chose DDN to complement the SGI system but they found that integrating between separate storage and compute vendors came with some unexpected overhead and effort. When they looked at what HPE Cray could offer for both the price performance was already there so bifurcating the deal between compute/storage wouldn’t make sense overall. “Cray met all our technical requirements for aggregate bandwidth and total usable capacity, so it was overall a win to have this integrated as well.”

The flash for metadata is needed because of NCAR’s mixed file system with many small and large files, which makes it difficult for standard file systems to keep up. “We found that having an SSD for metadata helps greatly, that’s another reason HPE added these to the configuration.” Elahi says. They have their own “Glade” file system at NCAR, which uses IBM Spectrum Scale (GPFS) but they will retire that entire system when the upgrade comes. It will be some heavy lifting to make the change, Elahi adds, but it will bring the system and its management up to date and make it more manageable.

On the software side, the Cray Operating System (SUSE Linux) based will be the foundation with Altair’s Accelerator Plus scheduled matched with PBS Pro as the workload manager. The system will include features new to NCAR, including the ability to tap to cloud resources, support for Docker and Singularity containers, and all the goodies that normally come with a Cray system (their programming environment and support for OpenMP 4.5 and 5.0 and MPI v3.1). On the Nvidia side they’ll have the HPC SDK with all the compilers and libraries for GPU computing.

Elahi says that most of their codes are not machine-spanning but they have several that will take over half the machine. These include complex multi-physics packages and some of their own home-grown climate models, including MPAS and low eddy simulations, among many others.

“We are inspired by leading R&D centers, such as the National Center of  Atmospheric Research (NCAR), in their research efforts of applying high performance computing to understanding earth systems, including severe  weather events and wildfires, that lead to informed decisions in earth’s  keeping people safe and preserving ecosystems,” said Bill Mannel, vice  president and general manager, HPC, at HPE. “We are honored to have  been selected by NCAR to build its next-generation supercomputer that  will tackle complex computational research for a range of geosciences  while also making it easier for researchers to store and make that data  accessible to help communities make smarter decisions.”