Adaptive Input/Output System upgrade further maximizes computation time on supercomputers
What good are supercomputers if you have to spend your whole time getting data in and out? Researchers at Oak Ridge National Laboratory (ORNL) are working to say goodbye to input/output (I/O) problems with their most recent upgrade of the Adaptive Input/Output System (ADIOS).
ADIOS grew out of a 2008 collaboration between the Oak Ridge Leadership Computing Facility (OLCF) and researchers from academia, industry, and national laboratories. Their goal was a system to get information in and out of a supercomputer efficiently.
“The measurement of success for us has always been ‘what percentage of time do you spend in I/O?’” said OLCF computational scientist Scott Klasky. ADIOS was inspired by Klasky’s work at the Princeton Plasma Physics Laboratory, where he noticed up to 30 percent of researchers’ computational time was spent reading and writing analysis files.
The open-source middleware is designed to help researchers maximize their allocations on leadership-class computing resources from wherever they may be. In essence, it creates more time for research by minimizing the time needed to read and write data to files, even if researchers are sending those files from thousands of miles away.
The previous release—version 1.2—improved usability by allowing users to construct new variables in their simulations while they run, simplifying the interface users work with and improving I/O performance. In fact, researchers using the middleware found the writing process substantially expedited, consuming only .06 percent of their computation time on average.
The biggest challenge for the newest release—version 1.3—was to improve reading efficiency, according to Qing Liu, a scientific researcher working in the Remote Data Analysis and Visualization branch at the National Institute for Computational Sciences and the leading developer of ADIOS.
“Our users were very happy that the writing was greatly improved [with version 1.2]. They could get 30 gigabytes per second for the writing performance, but when they tried to read, the performance was much lower,” he said. “There was a huge gap between writing and reading performance.”
Ray Grout, a researcher at the National Renewable Energy Laboratory, was one of the first people to test the latest update, using the S3D combustion code to study turbulent flows reacting to one another. Grout noted a huge increase in reading performance.
Klasky also added that ADIOS 1.4 will continue grass-root collaboration from computer science researchers to greatly reduce the problem of coping with large data on high-performance machines.
In addition to Klasky, Liu, and Grout, ADIOS 1.3 collaborators include William Tang, C.S. Chang, Weixing Wang, Stephane Ethier, and Zhihong Lin of the Princeton Plasma Physics Laboratory; Norbert Podhorszki, Jeremy Logan, Sean Ahern, Luis Chacon, Roselyne Tchoua, and Ricky Kendall of ORNL; Jackie Chen, Kenneth Moreland, Jay Lofstead, Ron Oldfield, and Todd Kordenbrock of Sandia National Laboratories; Xiaosong Ma, Nagiza Samatova, and Sriram Lakshminarasimh of North Carolina State University; Garth Gibson and Milo Polte of Carnegie Mellon University; Karsten Schwan and Greg Eisenhauer of Georgia Tech; Hasan Abbasi of the University of Tennessee–Knoxville; Seung-Hoe Ku of New York University; John Wu, Arie Shoshani, and Jinoh Kim of Lawrence Berkeley National Laboratory; and Julian Cummings of the California Institute of Technology.
For more information on ADIOS or to download the source, please visit https://www.olcf.ornl.gov/center-projects/adios/ —By Eric Gedenk
