New computer cluster powers advanced climate research for DOE’s Atmospheric Radiation Measurement user facility

The US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL) has launched a new, much more powerful computer system to support climate data research conducted by the DOE’s Atmospheric Radiation Measurement (ARM) user facility.

Procured and managed by the National Center for Computational Sciences (NCCS) at ORNL, the Cumulus-2 cluster is a high-performance computing (HPC) system from Dell that boasts 16,384 processing cores—easily surpassing ARM’s previous Stratus (1,080 cores) and Cumulus‑1 (4,032 cores) systems. The new system enables ARM to offer the HPC infrastructure needed to support next-generation atmospheric model simulations, petascale data storage, big-data analytics, and machine learning for atmospheric and climate science research.

“This new cluster will greatly accelerate processing speeds for simulations and boost capabilities to interpret ARM’s storehouse of data,” said Giri Prakash, director of the ARM Data Center at ORNL. “Cumulus-2 will offer roughly 4 times the power of Cumulus-1.”

Combined with its 4-petabyte parallel file system for expanded data storage, Cumulus-2’s increased computational power will further propel ARM’s mission “to advance a robust predictive understanding of Earth’s climate and environmental systems and to inform the development of sustainable solutions to the nation’s energy and environmental challenges.”

Formed in 1986 under DOE’s Biological and Environmental Research program, ARM’s primary task was to establish a series of instrument platforms to record atmospheric data for studying the Earth’s climate. ARM continues to collect this data from three atmospheric observatories across North America along with data gathered by mobile laboratories around the world in a partnership that spans nine national laboratories, including ORNL. With Cumulus-2, the user facility now provides modern HPC resources to atmospheric and climate researchers whose approved projects use ARM’s observational data.

“The majority of ARM users are from other labs, universities, and global research organizations. Last year, there were around 1,000 unique scientific users who produced over 200 journal articles. What the ARM Data Center team does at ORNL is the key for those scientists to access the data,” said Prakash, who also serves as section head for Earth System Informatics and Data Discovery in ORNL’s Environmental Sciences Division.

Prakash manages the data generated from ARM’s 450+ instruments that collect atmospheric phenomena. With 3,000–4,000 data streams coming in from facilities each day, Prakash’s team collects, processes, distributes, and archives the information—usually making it available to users within 24 hours. When he started his position in 2001, there were 20 terabytes of data collected from the instruments. By 2010, it grew to 200 terabytes, and from 2010 to 2016, it reached its first petabyte. Since December of 2016, they have amassed 3 petabytes of data, mostly from high-resolution instruments and 11,000+ data products.

“As part of the premier atmospheric facility at ORNL, we’re providing cutting-edge data and computing capabilities to support science use cases. We continue to improve with the mission to give a better user experience to our scientists,” Prakash said.

In addition to Prakash’s data management, the NCCS provides extensive support from its Operations and Systems Section to keep Cumulus-2 up and running—from hardware to software. Ryan Prout, an HPC data analytics engineer, serves as program manager; Elton Cranfill, group lead for HPC Clusters, manages the compute infrastructure; Dustin Leverman, group lead for HPC Storage Systems, manages the data infrastructure; and Brian Zachary, group lead for HPC Networking and Infrastructure, manages the network infrastructure.

At right: Giri Prakash, director of the data center for DOE’s Atmospheric Radiation Measurement (ARM) user facility, and Ryan Prout, HPC data analytics engineer, NCCS, with ARM’s Cumulus-2 system at ORNL. Image: Carlos Jones/ORNL

“The NCCS team led by Ryan Prout was key to the design and deployment of the Cumulus-2 architecture within the NCCS computing ecosystem. I would like to extend my sincere appreciation for their hard work and technical expertise in bringing this cluster online,” Prakash said.

A significant ARM project that uses Cumulus-2 is the Large-Eddy Simulation (LES) ARM Symbiotic Simulation and Observation (LASSO) effort. LASSO enhances ARM observations by using LES modeling to provide context for the data and a self-consistent representation of the atmosphere surrounding ARM’s observations. Current LASSO LES libraries include simulations at ARM’s Southern Great Plains atmospheric observatory and the Cloud, Aerosol, and Complex Terrain Interactions field campaign in Argentina.

“We’re integrating high-resolution modeling with the ARM observations so we can add value to the ARM products and provide users with new ways to conceptualize and use the data,” said William Gustafson, an atmospheric scientist at Pacific Northwest National Laboratory and principal investigator for LASSO.

For example, the models could help resolve bubbles of air near the ground surface to see how they turn into shallow convection clouds.

“These high-resolution simulations can provide insight into connections between different observations,” Gustafson said. “We don’t always know how they’re connected. Modeling gives us a plausible realization of what’s happening between those observations.”

UT-Battelle LLC manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit