Users of Oak Ridge National Laboratory’s (ORNL’s) computing systems will soon have access to a new data storage system managed by the National Center for Computational Sciences (NCCS). Themis, an IBM Spectrum Archive storage system, will offer users a flexible storage option in the Open security enclave and expand the range of data storage options available at ORNL.
Users of the Oak Ridge Leadership Computing Facility’s (OLCF’s) computing resources currently have two storage options depending on how quickly and easily they need to access their data. The first option is an IBM Spectrum Scale file system called Alpine, which stores hot data (i.e., scratch data that must be accessed immediately for research or analysis) on disk for up to 90 days. The second option is the High-Performance Storage System (HPSS), which holds cold data that must be preserved for years—or sometimes forever—on tape. The HPSS also serves as ORNL’s archival mass-storage resource and sits in the Moderate security enclave.
However, not all data needs meet those criteria. By combining medium-term, warm storage on disk with longer-term, archival storage on tape through IBM’s Spectrum Scale and Spectrum Archive systems, Themis offers an option for data that must be accessed quickly for longer than 90 days but doesn’t need to be preserved on tape indefinitely.
“The Spectrum Scale disk component of the system is designed to provide a middle ground for users that don’t necessarily need their data kept forever but don’t need it to live on the scratch file system either. At the same time, the tape component provides an archival resource in NCCS Open that supports cold data,” says Brenna Miller, the HPC (high-performance computing) systems engineer who leads the system’s deployment. “Themis addresses those different use cases in a single solution.”
The two-tier design also provides the option to customize the storage solution based on a program’s needs. James “Jake” Wynne, an HPC Linux systems engineer, has been working on the policy engine for Themis—the mechanism that executes the rules that determine how data is handled throughout its lifetime.
“A user will write a file, and we write rules that determine what to do with the data based on file age, size, or some other attribute that the policy engine will then execute. It gives us the ability to define rules for a user, such as, ‘Never move these files to tape,’ or, ‘Automatically move these files to tape.’ It gives us precise control over where data lives at any given point in time and in the future.”
Beyond pure storage, Themis offers a new capability for data sharing within NCCS and ORNL as well as with other DOE laboratories and external organizations.
“Themis has been designed to facilitate sharing of data between projects,” says Miller. “Its location in the NCCS Open security enclave removes a hurdle for data-centric projects.”
A tried and true system
Work on Themis began in the summer of 2021, and Miller drew on previous experience with the IBM file system during initial planning.
“Concrete planning for Themis started with identifying user needs and then looking at what technologies we are aware of and have solid experience with. We knew the Spectrum Scale file system would provide a well-suited instrument for dealing with data that’s more frequently accessed. And we did an evaluation of the Spectrum Archive stack the year before, so we had some idea that this would meet our needs.”
Jake Wynne, center, and Brenna Miller stand next to Themis, left. The new storage system offers users a flexible storage option in the Open security enclave.
Looking ahead
Miller and Wynne have been doing the hands-on work on Themis since early 2022 and are preparing it for its first users this autumn. While the system is still in the final stages of review and acceptance, the team is already working on additional features to improve the user experience. For example, Wynne is developing a tool to quickly provide information about their data (e.g., the file name will flash red if it exceeds the 20 terabyte size limit). The tool also color codes a data file based on where it is being stored because the file location determines whether it can be retrieved immediately or whether there will be a delay.
“This tool I’m developing will provide the user with some insight into where their data lives. It will color any file that lives on disk green—it’s nice and fast to get to. Any file that lives on tape will be colored red so the user can expect to wait a long time to recall that,” he explains. “This has given me a good opportunity to explore different technologies as well as tying into the Spectrum toolkit programmatically.”
Miller and Wynne are excited about Themis’s capabilities, which complement the OLCF’s existing world-class storage options.
“Since Themis is in the Open security enclave, it offers ease of access for certain NCCS users and flexibility in data retention through the policy engine,” says Miller.
“Themis fills a niche in storage—a bridge between warm and cold data—and combines it with the performance of Spectrum Scale, which lets users do science more quickly,” says Wynne.
The OLCF is a DOE Office of Science user facility.
UT-Battelle LLC manages Oak Ridge National Laboratory for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
