The Department of Energy’s Oak Ridge National Laboratory recently upgraded its Scalable Protected Infrastructure (SPI) for CITADEL — the framework of security protocols that protects sensitive data on systems at the Oak Ridge Leadership Computing Facility (OLCF) — by expanding the facility’s Arx2 file storage system.
Arx2 was originally installed in December 2024 with a storage capacity of 18 petabytes, but the high-performance file system received a big performance boost in August 2025 with the addition of a cabinet from the retired Alpine 2 storage system, which served the now-dismantled Summit supercomputer. Engineers from the National Center for Computational Sciences at ORNL integrated some of Alpine 2’s network-shared disks into Arx2’s General Parallel File System cluster, which allows multiple computers to access the same data simultaneously, and added the required enclosures and switches. These additions have increased Arx2’s storage to 32 petabytes and boosted read and write speeds by 50 percent for both its flash storage and higher-capacity disks.
Arx2’s expanded capacity and faster performance enable large, complex science projects that use protected data to better leverage the Frontier supercomputer’s exascale computing power (a quintillion or more calculations per second). Frontier has 9,856 compute nodes, each one consisting of a single 64-core AMD EPYC CPU and four AMD Instinct M1250X GPUs.
“We’re able to run any Frontier node in SPI mode, with a unique prologue, and point it to Arx2 so that it stores it on our secure file system. So, we can scale up to large jobs that still point to our secure file system instead of having a very small, dedicated, 100% separate enclave,” said Gina Jaklic, an NCCS storage system engineer and the subject matter expert for Arx2. “Our ability to do this securely across any node on Frontier is unique.”
Gina Jaklic, NCCS storage system engineer and the subject matter expert for the Arx2 secure storage system, oversaw its expansion with the addition of a cabinet from the retired Alpine 2 storage system, which served the now-dismantled Summit supercomputer. Photo: Carlos Jones, ORNL
The SPI combination of Arx2 and Frontier also makes the OLCF the only DOE Office of Science facility that offers such levels of security and computing power for open-science projects that use regulated data.
“Although other high-performance computing centers operate secure enclaves, our SPI serves as a fully integrated architecture that allows researchers access to Frontier-level performance while also meeting compliance standards for sensitive data such as protected health information,” said Josh Cunningham, protected-data manager for the OLCF.
Datasets with protected health information (PHI) may include identifiable patient records such as medical histories or billing information. PHI is protected under federal law by the Health Insurance Portability and Accountability Act and must be kept private, which requires stringent security protocols for its use in research projects. ORNL’s CITADEL framework not only protects such data from cyberattacks but also ensures that it cannot be accessed by other researchers or used by other projects.
The OLCF is currently hosting two research efforts that use protected patient data on Frontier. In collaboration with the National Cancer Institute, ORNL researchers use CITADEL to train natural language processing models that classify cancer pathology reports up to 18 times faster than traditional methods. And, in a project that originated on the OLCF’s Summit supercomputer, the Department of Veterans Affairs’ Million Veteran Program uses genomic and health data from over a million veterans — the largest biobank of its kind — to conduct population-level genome-wide association studies.
“Both of these projects depend on access controls and data storage protocols that comply with regulations surrounding protected health data. Furthermore, both projects demand massive computing power. Arx2 makes that balance possible for researchers,” said Cunningham.
With its ability to encrypt data from end to end with an external key manager, Arx2 allows OLCF users to work with and store different types of protected data beyond PHI, including personally identifiable information and data protected under International Traffic in Arms Regulations. Even the storage system’s cabinets are secured by lock and key to prevent anyone from physically accessing the nodes.
For added security, if a job is using Arx2, then the Frontier compute nodes allocated to that job are run in a more secure mode and on a different batch partition workload than those used by other jobs. When one of those secure jobs is complete, the nodes used are fully rebooted to ensure that any lingering data is wiped from memory.
These protocols and physical upgrades make Arx2 a highly secure storage counterpart for Frontier and for upcoming systems such as Discovery.
“Just as its predecessor Arx served Summit and then Frontier, Arx2 is designed as a forward-looking architecture that can evolve alongside our next generation of systems,” Cunningham said. “Our HPC Storage and Archive group has designed Arx2 as a modular system that can expand our SPI capability as new machines come online to ensure continued support for projects that need an extra level of data protection.”
The OLCF is a DOE Office of Science user facility located at ORNL.
UT-Battelle manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit energy.gov/science.
