Exascale systems are expensive but for labs retrofitting existing facilities for novel cooling, the compute, storage, network, and software are only the beginning of high costs.
When exascale supercomputers are announced, everyone tends to focus on the cost and capabilities of the machines themselves but facilities-driven power and cooling, especially at exascale, are a large chunk of budgets, both from a construction and operational perspective.
Consider Los Alamos National Laboratory with an existing large fleet of classified and unclassified systems housed at its Strategic Computing Center. The lab already has a pre-exascale machine, Trinity, in the facility and will soon be adding an actual exascale-capable machine, the Crossroads supercomputer, which will be around four times more powerful than Trinity.
Getting ready for Crossroads from a power and cooling standpoint has taken massive investment, around $100 million projected in 2018 when the budget request was placed. The project is now complete, earning an award this week from the U.S. Secretary of Energy for finishing the revamp ten months early and $20 million under budget.
The power and cooling construction project, called ECCCE (Exascale Class Computing Cooling Equipment), sets the stage for the forthcoming HPE/Cray Crossroads machine and will serve as the facilities basis for future machines housed in the 300,000 square-foot building, which cost $91 million to build over twenty years ago.
The installed cooling towers and equipment for the Exascale Class Computer Cooling Equipment Project at Los Alamos National Laboratory.
ECCE construction involved building new cooling towers and other structures for the center. The goal was to double the warm water cooling capabilities and pave the way for an additional fleet of systems beyond just Crossroads, all of which support the lab’s NNSA directives.
ECCE’s open-cell cooling towers chill the water via evaporation and according to the lab’s goals, are set to provide a minimum of 13.4 metawatts of additional warm water cooling via five additional open cooled towers just north of where the existing towers sit. This involves expanding large diameter pipes around the building and adding more pumps and heat exchangers. The original nearly $100 million budget figure also factored in a slew of electrical and mechanical gear to support this new loop of cooling capabilities.
“The ECCCE project required bringing 5,200 tons of cooling capacity to our strategic computing center,” says Kathye Segala, Associate Laboratory Director for Capital Projects at Los Alamos. “This successful project greatly improved the Laboratory’s high-performance computing capability and, by extension, our nation’s nuclear deterrent, well into the future.
LANL2 Installed tower piping, process piping, heat exchangers, and basket strainers for the Exascale Class Computer Cooling Equipment Project at Los Alamos National Laboratory.
Other national labs are making similar preparations for exascale machines or already made the investments during their pre-exascale system installations, including Oak Ridge National Lab, which made its warm water cooling changes in advance of the Summit supercomputer’s arrival and has had to add more power coming into the system to power Frontier.