Cray XC50 Accelerates Astrophysics In Japan

Researchers for centuries have relied on observational and theoretical astronomy for studying the stars, using telescopes and mathematical calculations to view planets and other objects, determine how they relate to each other, delve into mysteries like black holes and dark matter, and put into perspective the Earth’s place in the universe. With the advances in compute technology over the past few decades, researchers can now more easily view outer space through the lens of ones and zeroes encapsulated in simulations, seeing things far away and large that would otherwise be invisible.

The National Astronomical Observatory of Japan (NAOJ) for much of this decade has relied on supercomputers from Cray to power their computational astronomy efforts. Now the research organization has a much more powerful system at its disposal. Cray has installed the next-generation supercomputer – the NS-05 “Aterui II”— which delivers three times the performance of its Aterui predecessor and is the most powerful system in the world dedicated to astrophysical calculations.

The powerful system, with a peak performance of just over 3 petaflops, is based on Cray’s XC50 supercomputer and is powered by 40,200 Intel Xeon Gold 6148 processor cores, with each 20-core chip running at 2.4 GHz. The system came online in early June.

It’s the third generation of Cray-based Aterui systems that have powered research at the NAOJ’s Center for Computational Astrophysics. From 2013 to mid-2014, the center used the initial Aterui, a system based on Cray’s XC30 supercomputer that had a peak performance of 502 teraflops, 24,192 cores of the eight-core Xeon E5-2670 processors and 94.25 terabytes of main memory.

As demand for the system grew, the NAOJ revamped Aterui. Researchers swapped out older CPUs for newer ones, upgrading them to 25,440 cores of the 12-core Xeon E5-2690 v3 chips and dropping the number of cabinets from eight to six. Main memory was increased to 135.6 TB and the peak performance jumped to 1.058 petaflops. The newer system was launched in October 2014.

The new massively parallel Aterui II dwarfs those systems, including running 2,101 of the “Skylake” Xeon SP processors and providing 385.9 TB of main memory. Cray launched the XC50 family of supercomputers in 2016, aimed them at high-performance simulation, analytics, and machine learning workloads. The air-cooled system is designed for flexibility – it can run not only Intel’s Xeon Scalable processors but also the Arm-based ThunderX2 from Cavium (now owned by Marvell) and Nvidia’s Tesla P100 GPU accelerators – and includes high-performance communication libraries and Cray’s custom Aries interconnect to drive communication between the processors.

The Aterui II supercomputer will enable the NAOJ to perform highly compute-intensive calculations and run high-resolution simulations for such models as the formation and evolution of the Milk Way galaxy and three-dimensional simulations of a supernova explosion. About 150 researchers will use the new system to calculate such jobs as the gravitational forces among 200 billion stars in the Milky Way, whereas before the stars would have to be collected in groups for other industry simulations.

“Our new Cray XC50 gives us the computational capability required to solve challenges that previous systems could not address and calculate astrophysical simulations in a more realistic way,” said Eiichiro Kokubo, project director for the Center for Computational Astrophysics at NAOJ.

Cray has seen its high-end supercomputer business ebb and flow over the past several quarters. The company in the latest financial quarter lost $25 million but saw revenue jump from $59 million the previous three months to $79.6 million. In a conference call in May, CEO Peter Ungaro the overall market contracted due to a number of factors, including uncertainty in government budgets with new administrations in multiple counties, a downturn in purchases in such verticals as energy and slowdown in decision making by customers. However, those areas are improving.

“The government budget landscape is beginning to settle and decision makers are better able to solidify their plans both over the short and longer term,” Ungaro said.

Prospects are improving in the United States, United Kingdom and European Union, he said, adding that the market in Asia is turning around quickly. The CEO noted that the company will build a XC50 supercomputer to the National Institutes for Quantum and Radiological Science and Technology in Japan. At more than 4 petaflops, the new system will be more than twice as powerful as the existing Helios Bullx cluster, which tops out at 1.5 petaflops.

Cray also over the past few years has been looking to expand its reach beyond the highly competitive supercomputer space and into the commercial market.