Advances in In Situ Processing Tie to Exascale Targets

Waiting for a simulation to complete before visualizing the results is often an unappealing prospect for researchers.

Verifying that output matches expectations early in a run helps prevent wasted computation time, which is particularly important on systems in high demand or when a limited allocation is availableIn addition, the growth in the ability to perform computation continues to outpace the growth in the ability to performantly store the results. The ability to analyze simulation output while it is still resident in memory, known as in situ processing, is appealing and sometimes necessary for researchers running large-scale simulations.

In light of this need, in situ processing has received more attention of late. The Scalable Environment for Scientific Exploration In Situ (SENSEI) project, lead by E. Wes Bethel of Lawrence Berkeley National Laboratory, is a notable entry into the field. The team  recently completed a study, to be presented at Supercomputing later this month, evaluating the performance impacts of running SENSEI alongside simulations. The largest scale simulation was a 1,048,576 MPI rank fluid dynamics simulation using PHASTA. This is believed to be the largest in situ MPI simulation ever performed.

“This is part of the exascale initiative,” project member Andrew Bauer of Kitware tells The Next Platform, “trying to get more accurate analysis and visualization, trying to get more scientific insight or more engineering insight into the types of problems that we’re trying to solve.”

The PHASTA simulation was able to make use of SENSEI to provide live visualization during the course of the simulation. Researchers were able to observe the airflow over an aircraft wing as the simulation ran. Furthermore, because PHASTA supports on-the-fly reconfiguration of input parameters, the researchers were able to manipulate the simulation live in order to determine the maximum aerodynamic performance configuration. “We’re using in-situ [processing] basically to try to make the machine more useful to the scientists and engineers, making sure they can get the information they want…as conveniently as possible,” Bauer says.

The large PHASTA spent 13% of the simulation time in in situ processing. In situ time for the other scales and simulation codes in the study varied, but the consistent theme was that in situ processing outperformed post hoc processing of results. This is due to both the immediate ability to visualize results as well as the cost of writing output to disk. The study reports in situ processing affords 3-4 times greater temporal resolution for analysis in a reactive flow multiphysics code compared to writing data for post hoc analysis. Some SENSEI-enabled runs of the computational cosmology package Nyx had shorter wall clock times than the baseline measure.

As many early-stage paradigms do, in situ processing faces a lack of interoperability. SENSEI attempts to address this issue by providing a generic data interface using the VTK data model. The data adaptor provides a translation between data structures used by the simulation and SENSEI’s data model. This allows simulation applications to make use of SENSEI’s data interface without code modification. Similarly, SENSEI’s visualization adaptor connects the data model to analysis code such as ParaView or VisIt. A lightweight bridge connects the two adaptors.

The ability to “write once, use anywhere” is an important goal of the SENSEI project, as it frees simulation developers from having to natively support numerous in situ analysis infrastructures. Analysis developers are also freed from having to commit to either choose a single infrastructure to support or to support multiple infrastructures. Making in situ processing more easily available to scientists and engineers, particularly at extreme scales, is a driving factor behind SENSEI.

As we approach exascale systems, the gap between FLOPS and IOPS will continue to grow. Scientists and engineers using these systems will be left with in situ processing as the only option  in many cases. The SENSEI team has shown that in situ processing can work at the million core scale, and they plan to keep ahead of users. “We want to make sure that everything is working smoothly when it gets to the user so they adopt this functionality and its new computing paradigm,” Bauer said. “If we want people to adopt … in-situ infrastructure for doing their analysis and visualization, we have to show that our tool scales at least as well as they need it to and hopefully beyond.”

SENSEI has shown itself to meet the goal of being a lightweight, portable, and flexible tool for in situ analysis and visualization for today’s fluid dynamics simulations. Other applications and use cases will be studied to improve SENSEI’s applicability as a generic framework.