The Adaptable IO System (ADIOS) provides a simple, flexible way for scientists to describe the data in their code that may need to be written, read, or processed outside of the running simulation. By providing an external to the code XML file describing the various elements, their types, and how you wish to process them this run, the routines in the host code (either Fortran or C) can transparently change how they process the data.
The in code IO routines were modeled after standard Fortran POSIX IO routines for simplicity and clarity. The additional complexity including organization into hierarchies, data type specifications, process grouping, and how to process the data is stored in an XML file that is read once on code startup. Based on the settings in this XML file, the data will be processed differently. For example, you could select MPI individual IO, MPI colletive IO, POSIX IO, an asynchronous IO technique, visualization engine, or even NULL for no output and cause the code to process the data differently without having to either change the source code or even recompile.
The real goal of this system is to give a level of adaptability such that the scientist can change how the IO in their code works simply by changing a single entry in the XML file and restarting the code. The ability to control at a per element basis and not just a data grouping such as a restart, diagnostic output, or analysis output makes this approach very flexible. Along with this detail level, a user can also just change which transport method is used for a data type such as a restart, analysis, or diagnostic write.
For the transport method implementer, the system provides a series of standard function calls to encode/decode data in the standardized .bp file format as well as “interactive” processing of the data by providing direct downcalls into the implementation for each data item written and also callbacks when processing a dodata stream once a data item has been identified along with its dimensions and a second callback once the data has been read giving the implementation the option to allocate memory and process the data as close to the data source as is reasonable.
Below are links for the Adios download and user manual. For additional support, please contact [email protected].
Download
Current stable version: ADIOS 1.13.1 (Download 1.13.1, manual)
Revision History
1.13.1 Release April 2018
- fix: zero size variable-blocks are supported by all transformations.
- fix: build issues with the profiling interface.
- fix: free temporary MPI communicators so that applications don’t run out of communicators when using ADIOS for many output steps.
- fix: build issues with Flexpath staging.
1.13.0 Release November 2017
- Added blosc compression transform by René Widera HZDR, Germany
- TAU now can profile ADIOS in applications, by Kevin Huck of the TAU team
- update to use SZ v1.4.11 in compression
- bpls -dD option to dump data in per-writer fashion (aka reading with writeblock selection)
- fix: bug fixes to new Flexpath staging method (MPI communicator, memory leaks)
- fix: bpls command-line allows for large integers to dump data from >2GB blocks
- support for JoinedArray, where per-writer data blocks are virtually merged into
a global array by ADIOS at read time for easier reading.
1.12.0 Release June 2017
- added LZ4 compression transform by René Widera HZDR, Germany
- added SZ compression transform
- Support for burst buffer on Summitdev@OLCF, Cori@NERSC
- see Chapter 16 in the manual
- New Flexpath staging method from Georgia Tech, faster and more reliable
- Performance tools interface API by Kevin Huck of the TAU team
- fix: cmake build broken since 1.11.0. Also include building
the test suite - fix: for >2GB block reads
- fix: time-aggregation works with writing multiple files each
contaning multiple steps - fix: python interface
1.11.1 Release January 2017
- fix: able to read from a BP file when the index itself is > 2GB
- fix: bpmeta, bpdump works correctly with index size > 2GB
1.11.0 Release November 2016
- Time aggregation (temporal aggregation of an output in memory)
see adios_set_time_aggregation() or the <time-aggregation> element in the XML syntax. - ZFP lossy compression transform method
- Python wrapper includes functions for:
- Selecting transforms and time aggregation: adios_set_transform()
- Time aggregation: adios_set_time_aggregation()
- Set maximum buffer size used by any ADIOS group: adios_set_max_buffer_size()
- Collect min/max statistics only by default
adios_declare_group() last argument type changed to be an option for statistics.
Options are: adios_stat_no, adios_stat_minmax, adios_stat_full, and adios_stat_default, which is minmax - Added functions to C API to detect available methods in the ADIOS installation
adios.h: adios_available_write_methods()
adios_read.h: adios_available_read_methods()
adios_transform_methods.h: adios_available_transform_methods()
adios_query.h: adios_available_query_methods() - Performance bug in MPI_AGGREGATE method in 1.9/1.10 fixed
- fix: bug building with hdf5 1.10
1.10.0 Release July 2016
- Updated Query API and Minmax, FastBit and Alacrity query methods
- Fortran API for inquiring selections
- ADIOS builds without first installing Mini-XML separately
- bprecover utility
- recover a BP file which has a damaged index data
- adios_group_size() optional now
- –without-mpi option in configure to build only the sequential libraries
- Python/Numpy wrapper improvements:
- Support both python 2 and python 3
- Read options with point and block selection
- Group management on reading
- Support auto completion with ipython
- fix: bpmeta does not skip any subfiles when used with threads
- fix: Better xml processing to allow for multiple text lines as parameters for a method
- fix: Builds on OS X, both clang and gcc supported
- fix: support adios_inq_var_stat() in streaming mode
1.9.0 Release December 2015
- Array attributes are supported
- e.g axes = {“X”,”Y”,”Z”}
- adios_define_attribute_byvalue()
- to define scalar attributes with floating point variables instead of strings for precision
- Update mode when appending to a file
- to add variables to last timestep instead of a new one
- Python/Numpy wrapper improvements:
- Numpy-style array notations
e.g, var[1:5, 2:10], var[1:5. :], var[:5,…] - Support for ADIOS write APIs
- Hint/docstring support
- Support for pip install and update
- Numpy-style array notations
- Added adios_version.h to provide release and file format versions
- Bug fixes
- fix: memory leak in POSIX method
- fix: adios_write() now accepts const * void data from C++ apps
- fix: Cray compiler support
- fix: reading of compressed, zero size arrays on some processes
- fix: scaling bugs in aggregate method writing > 2GB per process or aggregating data into a file over 4GB
1.8.0 Release December 2014
- Query API
- extends the read API with queries
- Staging over WAN (wide-area-network)
- ICEE method (requires FLEXPATH)
- skeldump utility
- to generate info and code from output data to replay the I/O pattern
- bpmeta utility
- generates metadata file (.bp) separately after writing the data using MPI_AGGREGATE method with metadata writing turned off
- I/O timing statistics and timing events can be collected
- New stage writer code for staged I/O, where output data (list of variables and their sizes) is changing at every timestep. See examples/stage_write_varying
- fix: staging with multiple streams allowed
- fix: parallel build (make -j <n>) completes without breaking
1.7.0 Release June 2014
- Support for more than 64k variables in a file
- File system topology aware I/O method for Titan@OLCF
- DataSpaces staging
- support for 64bit dimension sizes
- support for more than three dimensions
- works on Bluegene/Q (DataSpaces+DIMES methods)
- can run as a service (dynamic connections)
- Additions to non-XML Write API:
- Support for the visualization schema
- adios_set_transform() to choose the transformation for a variable
- Usability improvements:
- CMake Module for find_package(ADIOS)
- adios_config -m to print available write/read methods
1.6.0 Release December 2013
- Transformations of data supported in file-based I/O
- lossless compression (zlib, bzip, szip)
- lossy compression (ISOBAR)
- precision-level-of-detail encoding (APLOD)
- Changes to Write API:
- variables are identified by full path at writing
- fix: all int functions return 0 as OK and !=0 on error
- Changes to Read API:
- Read API extensions to get information about the visualization meshes defined in a file
- leading / in path names is not enforced
- New I/O method for Bluegene/Q called “BGQ” configure with the option –with-bgq
- Removed performance bottleneck in metadata operations when writing/reading thousands of variables.
- fix: one can build ADIOS separately from the source with automake
1.5.0 Release June 2013
- Changes to Write API:
- adios_init() and adios_init_noxml() has MPI_Comm argument
- adios_open() has MPI_comm argument instead of void * argument
- Changes to Read API:
- adios_read_open_stream() obsolete and now it’s called adios_read_open() indicating that it’s used for both files and staged data.
- New staging methods:
- DIMES
- FLEXPATH
- CMAKE build files (besides Automake files)
- New write method VAR_MERGE for spatial aggregation of small per-process-output into larger chunks. It improves both write and read performance for such applications.
- fix: segfault in adios_inq_var_blockinfo()
- fix: endianness independence ()
- fix: in adios_inq_var_stat() for getting statistics (avg, std.dev)
- fix: backward compatibility in reading old BP files containing scalars over time (a 1D array)
1.4.1 Release December 2012
- aggregated file reader method (from 1.3.1): use ADIOS_READ_METHOD_BP_AGGREGATE
- memory limitation specified by user for chunked reading now taken into account in file reading
- stage writer example code for staged I/O see examples/stage_write
- code coupling example code for file-based/staged coupling see examples/coupling
- bp2h5 supports converting complex and double complex types
- new adios_write_byid() function to write multiple-blocks of the same global array from a process
- fix: F90 modules: adios_write, adios_schedule_read
- fix: invalid “out of bound” errors when reading multiple steps of multi-dimensional arrays
- fix: double-free bug in support of old read API in Fortran
- fix: backward compatibility of old read API to read multi-group files (fixed “invalid variable id…” errors)
1.4.0 Released July 2012
- new read API for staging methods with step-by-step processing, also with non-blocking and chunking APIs
- visualization schema added to ADIOS XML format
- skel: I/O skeleton generator and evaluation tools
- unified error/debug logging
- hidden attributes (version, create/update times)
- Java and Numpy bindings
- F90 modules adios_write_mod and adios_read_mod to check syntax at compile time
1.3.1 Released November 2011
- fix: non-compliant MPI collective calls
- fix: MPI_AMR method caused MPI_Finalize issues on some clusters
- fix: histogram calculation: freeing memory too early
1.3 Released July 2011
- New read method BP_STAGED for reading files with improved performance
- Changes in MPI_AMR and MPI_LUSTRE to support default parameters. Parameters for MPI_LUSTRE and MPI_AMR in XML file are not mandatory any more.
1.2.1 Released August 2010
- Bug fix in read API (to read arrays written into a file over several timesteps but without specifying time dimension)
- Added DIMES method from Rutgers for tight-coupling applications
1.2 Released July 2010
- XML and non-XML write APIs are available
- More statistics (min/max/avg/std.dev/histograms) available without overhead in write performance
- Added MPI_AMR method for advanced mesh refinement codes and for aggregated writing (N procs to P writers to M disks)
- Added support for subfiles
- POSIX method many-file output can be read as one file
- MPI_AMR method uses subfiles to improve write performance
- Added NetCDF-4 transport method
- Asynchronous, staging methods on Jaguar XT5 at ORNL
- DataTap from Georgia Tech
- DataSpaces from Rutgers
- NSSI from Sandia
- MPI_LUSTRE method for best performance on Lustre file system
1.0.1 Released on December 2009
- fix: builds read API on Mac (was broken at utils/bpdump)
- fix: Fortran written BP files are correctly read by Fortran readers
- added adios_mpi_stripe2 method for improved performance on Lustre file system
1.0 Released on November 2009
Requirements and optional features
- ADIOS requires MPI and MPI-IO.
- Python (required)
The XML processing utility utils/gpp/gpp.py is a code written in python using xml.dom.minidom. It is used to generate C or Fortran code from the XML configuration files that can be included in the application source code. Examples and tests will not build without Python. - A Fortran 90 compiler (optional)
The Fortran API is built only if a Fortran 90 compiler is available.
Use--disable-fortran
at configuration time to disable this feature. - DataSpaces staging (optional)
- The DataSpaces model provides a separate server running on separate compute nodes, into/from which data can be written/read with a geometrical (3D) abstraction. It is an efficient way to stage data from one application to another in an asynchronous (and very fast) way. Multiple steps of data outputs can be stored, limited only by the available memory. DataSpaces can be downloaded from https://www.dataspaces.org.
- DataSpaces is supported on Cray Gemini, Portals and Infiniband networks.
- Flexpath staging (optional)
- Flexpath is an asynchronous data transport method built to ensure scalable I/O through the use of staging areas. It is built on top of the EVPath event-driven messaging library, which allows for the creation of arbitrary network overlays. Flexpath also works on top of several popular high-end network interfaces, such as Infiniband, Portals, and Gemini.
- EVPath acquisition instructions are at the EVPath web site, or in the ADIOS manual under the installation guide.
- FastBit indexing library (optional)
- The Query API in ADIOS requires FastBit. FastBit (https://sdm.lbl.gov/fastbit) can be downloaded from a public SVN repository: svn co https://codeforge.lbl.gov/anonscm/fastbit/trunk
- Numpy (optional)
- If you have Numpy, you can build ADIOS read/write modules for Numpy separatly after building ADIOS. See details in the Installation section of the User’s Manual.
- Matlab (optional)
- If you have Matlab, you can build the ADIOS reader for Matlab separatly after building ADIOS. See details in the Matlab section of the INSTALL file before configuring adios.
- HDF5 (optional)
- The bp2h5 converter utility to HDF5 format is built only if a HDF5 library is available. Currently ADIOS uses the 1.6 version of the HDF5 API but it can be built and used with the 1.8.x version of the HDF5 library too. Use the option
--with hdf5=
to configure ADIOS. - HDF5 can be downloaded from https://www.hdfgroup.org/downloads
- The bp2h5 converter utility to HDF5 format is built only if a HDF5 library is available. Currently ADIOS uses the 1.6 version of the HDF5 API but it can be built and used with the 1.8.x version of the HDF5 library too. Use the option
- NetCDF (optional)
- The bp2ncd converter utility to NetCDF format is built only if NetCDF is available. ADIOS can use a sequential build of the NetCDF-3 or NetCDF-4 library. Use the option –with-netcdf= or ensure that the NETCDF_DIR environment variable is set before configuring ADIOS.
- NetCDF version 4.3.2 or the legacy version 3.6.2 can be downloaded from https://www.unidata.ucar.edu/downloads/netcdf
- Parallel HDF5 (optional)
- The transport method writing files in the Parallel HDF5 format is built only if a parallel version of the HDF5 library is available. You need to use the option with
phdf5=
to build this transport method.If you define Parallel HDF5 and do not define serial HDF5, then bp2h5 will be built with the parallel library.Note that if you build this transport method, ADIOS will depend on PHDF5 when you link any application with ADIOS even if you the application does not intend to use this method.If you have problems compiling ADIOS with PHDF5 due to missing flags or libraries, you can define them using--with-phdf5-incdir=
,--with-phdf5-libdir=
and--with-phdf5-libs=
- The transport method writing files in the Parallel HDF5 format is built only if a parallel version of the HDF5 library is available. You need to use the option with
- NetCDF-4 Parallel (optional)
- The NC4 transport method writes files using the NetCDF-4 library which in turn is based on the parallel HDF5 library. You need to use the option
-- with nc4par=
to build this transport method. Also, you need the parallel HDF5 library. - NetCDF version 4.0.1 can be downloaded from https://www.unidata.ucar.edu/downloads/netcdf
- You need to build NetCDF-4 with a parallel HDF5 library as described at https://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install/parallel.html
- The NC4 transport method writes files using the NetCDF-4 library which in turn is based on the parallel HDF5 library. You need to use the option
User’s Manual & Installation
User’s manual: ADIOS-UsersManual-1.13.1.pdf
Skel manual: skel-doc-1.6.0.pdf
Visualization schema in ADIOS: ADIOS-VisualizationSchema-1.1.pdf
Developer’s manual: ADIOS-DevManual-1.6.0.pdf
Archived Manuals
- 1.13.1 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.13.1.pdf
- 1.13.0 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.13.0.pdf
- 1.9.0 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.9.0.pdf
- 1.8.0 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.8.0.pdf
- 1.7.0 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.7.0.pdf
- 1.3.1 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.3.1.pdf
- 1.3 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.3.pdf
- 1.2.1 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.2.1.pdf
- 1.2 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.2.pdf
- 1.0 User’s Manual/Installation: https://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.0.pdf
Press
ADIOS Ignites Combustion Simulations
https://www.hpcwire.com/features/ADIOS-Ignites-Combustion-Simulations-67321602.html
Fusion Gets Faster
https://www.hpcwire.com/features/Fusion-Gets-Faster-51820167.html?viewAll=y
Researchers Conduct Breakthrough Fusion Simulation
https://www.hpcwire.com/offthewire/Researchers_Conduct_Breakthrough_Fusion_Simulation.html
Publications
1. Liu, Q., Logan, J., Tian, Y., Abbasi, H., Podhorszki, N., Choi, J. Y., Klasky, S., Tchoua, R., Lofstead, J., Oldfield, R., Parashar, M., Samatova, N., Schwan, K., Shoshani, A., Wolf, M., Wu, K. and Yu, W. (2013), Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency Computation: Practice and Experience. doi: 10.1002/cpe.3125
https://onlinelibrary.wiley.com/doi/10.1002/cpe.3125/abstract
2. Cummings, Klasky, Podhorszki, Barreto, Lofstead, Schwan, Docan, Parashar, Sim, Shoshani, “EFFIS: and End-to-end Framework for Fusion Integrated Simulation”, Proceedings of the 18th Euromicro Conference on Parallel, Distributed and Network-based Processing, PDP 2010, Pisa, Italy, February 17-19, 2010
https://www.researchgate.net/publication/221392706_EFFIS_An_End-to-end_Framework_for_Fusion_Integrated_Simulation
3. N. Podhorszki, S. Klasky, Q. Liu, C. Docan, M. Parashar, H. Abbasi, J. Lofstead, K. Schwan, M. Wolf, F. Zheng, J. Cummings. “Plasma fusion code coupling using scalable I/O services and scientific workflows“. In Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science (WORKS ’09). ACM, New York, NY, USA, , Article 8 , 9 pages. DOI=10.1145/1645164.1645172
https://doi.acm.org/10.1145/1645164.1645172
4. Bent, Gibson, Klasky, Lofstead, Parashar, Polte, Schwan, “… And eat it too: High read performance in write-optimized HPC I/O middlware file formats”. In Proceedings of the 4th Annual Workshop on Petascale Data Storage (2009), pp. 21-25
https://repository.cmu.edu/cgi/viewcontent.cgi?article=1074&context=pdl
5. Abbasi, H., Wolf, M., Eisenhauer, G., Klasky, S., Schwan, K., and Zheng, F. 2009. DataStager: scalable data staging services for petascale applications. In Proceedings of the 18th ACM international Symposium on High Performance Distributed Computing (Garching, Germany, June 11 – 13, 2009). HPDC ‘09. ACM, New York, NY, 39-48.
https://dl.acm.org/citation.cfm?id=1551618
6. Hasan Abbasi, Jay F. Lofstead, Fang Zheng, Karsten Schwan, Matthew Wolf, Scott Klasky: Extending I/O through high performance data services. CLUSTER 2009: 1-10
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.165.2623&rep=rep1&type=pdf