The OLCF provides a comprehensive suite of hardware and software resources for the creation, manipulation, and retention of scientific data. A summary of the OLCF’s Data Management Policy is presented below.
Data Storage Areas
The OLCF provides an array of data storage platforms, each designed with a particular purpose in mind. Storage areas are broadly divided into two categories: those intended for user data and those intended for project data. Within each of the two categories, we provide home areas (for routine activities), work areas (for computationally-intensive activities) and archive areas (for archival storage).
|User||Home||Routine user activities involving frequently-accessed data.|
|Work||Computationally-intensive activities within batch jobs.|
|Archive||Long-term, archival storage for infrequently-accessed user data.|
|Project||Home||Shared project data of a general nature, e.g., source code.|
|Work||Large shared project datasets needed as inputs within batch jobs.|
|Archive||Long-term, archival storage for infrequently-accessed project data.|
Data Retention, Purge, & Quota Summary
Users must agree to the full Data Management Policy as part of their account application. The “Data Retention, Purge, & Quotas” section is useful and is summarized below. It applies to all projects with allocations on OLCF systems.
||HPSS||2TB/2k files||No||None||3 months|
||Lustre||2 TB||No||None||3 months|
||HPSS||100TB/100k files||No||None||3 months|
|Area||The general name of storage area.|
|Nickname||The branded name given to the storage area.|
|Path||The path (symlink) to the storage area’s directory.|
|Type||The underlying software technology supporting the storage area.|
|Quota||The limits placed on total number of bytes and/or files in the storage area.|
|Backups||States if the data is automatically duplicated for disaster recovery purposes.|
|Purge||Period of time, post-file-creation, after which a file will be marked as eligible for permanent deletion.|
|Retention||Period of time, post-account-deactivation or post-project-end, after which data will be marked as eligible for permanent deletion.|
The OLCF provides a number of hardware resources for the physical storage and management of large amounts of scientific data.
The OLCF’s center-wide Lustre file system, called Spider, is the operational work file system for most OLCF computational resources. As an extremely high-performance system, Spider has over 26,000 clients, providing 10.7 petabytes of disk space and can move data at more than 240 gigabytes a second. For more information, see the Spider page within our support documentation.
HPSS is the archival mass-storage resource at ORNL and consists of robotic tape and disk storage components, Linux servers, and associated software. Incoming data is written to disk and later migrated to tape for long term archival. As storage, network, and computing technologies continue to change, ORNL’s storage system evolves to take advantage of new equipment that is both more capable and more cost-effective. For more information on HPSS allocations and use, see the Understanding HPSS Storage Allocations page or the Transferring Data with HSI and HTAR page within our support documentation.
The OLCF provides an array of software tools for managing large amounts of scientific data. All relevant software packages are listed under the Software: Data Management section of our support documentation.
Highlighted below are some OLCF-developed data management tools of particular interest:
The Adaptable I/O System (ADIOS) library, developed at OLCF, is a flexible method for applications to transparently manage data I/O. Using ADIOS, applications can change their I/O patterns and methods (such as MPI-IO or POSIX I/O) by simply editing a configuration file. This allows applications to optimize their I/O workload for different computing systems utilizing different I/O subsystems and filesystems. For more information on ADIOS, see the ADIOS project page or the ADIOS software page within our support documentation.
The Lustre User Toolkit, developed at OLCF, is an API for interacting with Lustre filesystems. LUT can also provide I/O timing information, allowing application authors to optimize their use of Lustre filesystems. OLCF has also made available lustre-optimized user utilities developed using LUT. The spdcp program is a fast, parallel tool to copy large datasets on and across Lustre filesystems. For more information on LibLUT or SPDCP, see the LibLUT software page or the SPDCP software page within our support documentation.
LustreDU was developed at the OLCF to give end-users the ability to view the last recorded size of a directory on Spider. The advantage of LustreDU is that the query does not hit the Lustre metadata servers as a normal `du` command would; rather, the query pulls from a seperate database and quickly returns the last known size of the directory. For more information, see the LustreDU software page in our support documentation.