titan

Up since 11/8/17 02:45 pm

eos

Up since 11/14/17 11:20 pm

rhea

Up since 10/17/17 05:40 pm

hpss

Up since 11/20/17 09:15 am

atlas1

Up since 11/15/17 07:25 am

atlas2

Up since 11/27/17 10:45 am
OLCF User Assistance Center

Can't find the information you need below? Need advice from a real person? We're here to help.

OLCF support consultants are available to respond to your emails and phone calls from 9:00 a.m. to 5:00 p.m. EST, Monday through Friday, exclusive of holidays. Emails received outside of regular support hours will be addressed the next business day.

Data Management Policy

See this article in context within the following user guides: Data | Policies
Note: This details an official policy of the OLCF, and must be agreed to by the following persons as a condition of access to or use of OLCF computational resources:

  • Principal Investigators (Non-Profit)
  • Principal Investigators (Industry)
  • All Users

Title: Data Management Policy
Version: 14.01

Introduction

The OLCF provides a comprehensive suite of hardware and software resources for the creation, manipulation, and retention of scientific data. This document comprises guidelines for acceptable use of those resources. It is an official policy of the OLCF, and as such, must be agreed to by relevant parties as a condition of access to and use of OLCF computational resources.

Data Storage Resources

The OLCF provides an array of data storage platforms, each designed with a particular purpose in mind. Storage areas are broadly divided into two categories: those intended for user data and those intended for project data. Within each of the two categories, we provide different sub-areas, each with an intended purpose:

Purpose Storage Area Path

Long-term data for routine access that is unrelated to a project User Home $HOME
Long-term data for archival access that is unrelated to a project User Archive /home/$USER
Long-term project data for routine access that’s shared with other project members Project Home /ccs/proj/[projid]
Short-term project data for fast, batch-job access that you don’t want to share Member Work $MEMBERWORK/[projid]
Short-term project data for fast, batch-job access that’s shared with other project members Project Work $PROJWORK/[projid]
Short-term project data for fast, batch-job access that’s shared with those outside your project World Work $WORLDWORK/[projid]
Long-term project data for archival access that’s shared with other project members Project Archive /proj/[projid]
User Home

Home directories for each user are NFS-mounted on all OLCF systems and are intended to store long-term, frequently-accessed user data. User Home areas are backed up on a daily basis. This file system does not generally provide the input/output (I/O) performance required by most compute jobs, and is not available to compute jobs on most systems. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

User Archive

The High Performance Storage System (HPSS) is the tape-archive storage system at the OLCF and is the storage technology that supports the User Archive areas. HPSS is intended for data that do not require day-to-day access. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

Project Home

Project Home directories are NFS-mounted on selected OLCF systems and are intended to store long-term, frequently-accessed data that is needed by all collaborating members of a project. Project Home areas are backed up on a daily basis. This file system does not generally provide the input/output (I/O) performance required by most compute jobs, and is not available to compute jobs on most systems. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

Member Work

Project members get an individual Member Work directory for each associated project; these reside in the center-wide, high-capacity Lustre® file system on large, fast disk areas intended for global (parallel) access to temporary/scratch storage. Member Work directories are provided commonly across all systems. Because of the scratch nature of the file system, it is not backed up and files are automatically purged on a regular basis. Files should not be retained in this file system for long, but rather should be migrated to Project Home or Project Archive space as soon as the files are not actively being used. If a file system associated with your Member Work directory is nearing capacity, the OLCF may contact you to request that you reduce the size of your Member Work directory. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

Project Work

Individual Project Work directories reside in the center-wide, high-capacity Lustre file system on large, fast disk areas intended for global (parallel) access to temporary/scratch storage. Project Work directories are provided commonly across most systems. Because of the scratch nature of the file system, it is not backed up. If a file system associated with Project Work storage is nearing capacity, the OLCF may contact the PI of the project to request that he or she reduce the size of the Project Work directory. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

World Work

Each project has a World Work directory that resides in the center-wide, high-capacity Lustre file system on large, fast disk areas intended for global (parallel) access to temporary/scratch storage. World Work directories are provided commonly across most systems. Because of the scratch nature of the file system, it is not backed up. If a file system associated with World Work storage is nearing capacity, the OLCF may contact the PI of the project to request that he or she reduce the size of the World Work directory. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

Project Archive

The High Performance Storage System (HPSS) is the tape-archive storage system at the OLCF and is the storage technology that supports the User Archive areas. HPSS is intended for data that do not require day-to-day access. Project Archive areas are shared between all users of the project. Users should not store data unrelated to OLCF projects on HPSS. Project members should also periodically review files and remove unneeded ones. See the section “Data Retention, Purge, & Quota Summary” for more details on applicable quotas, backups, purge, and retention timeframes.

Local Scratch Storage

A large, fast disk area intended for parallel access to temporary storage in the form of scratch directories may be provided on a limited number of systems. This area is local to a specific system. This directory is, for example, intended to hold output generated by a user’s job. Because of the scratch nature of the file system, it is not backed up and files are automatically purged on a regular basis. Files should not be retained in this file system and should be migrated to archival storage as soon as the files are not actively being used. Quotas may be instituted on a machine-by-machine basis if deemed necessary.

Data Retention, Purge, & Quotas
Summary

The following table details quota, backup, purge, and retention information for each user-centric and project-centric storage area available at the OLCF.

User-Centric Storage Areas
Area Path Type Permissions Quota Backups Purged Retention
User Home $HOME NFS User-controlled 10 GB Yes No 90 days
User Archive /home/$USER HPSS User-controlled 2 TB [1] No No 90 days
Project-Centric Storage Areas
Area Path Type Permissions Quota Backups Purged Retention
Project Home /ccs/proj/[projid] NFS 770 50 GB Yes No 90 days
Member Work $MEMBERWORK/[projid] Lustre® 700 [2] 10 TB No 14 days     [4]
Project Work $PROJWORK/[projid] Lustre® 770 100 TB No 90 days     [4]
World Work $WORLDWORK/[projid] Lustre® 775 10 TB No 90 days     [4]
Project Archive /proj/[projid] HPSS 770 100 TB [3] No No 90 days
Area The general name of storage area.
Path The path (symlink) to the storage area’s directory.
Type The underlying software technology supporting the storage area.
Permissions UNIX Permissions enforced on the storage area’s top-level directory.
Quota The limits placed on total number of bytes and/or files in the storage area.
Backups States if the data is automatically duplicated for disaster recovery purposes.
Purged Period of time, post-file-access, after which a file will be marked as eligible for permanent deletion.
Retention Period of time, post-account-deactivation or post-project-end, after which data will be marked as eligible for permanent deletion.
Important! Files within “Work” directories (i.e., Member Work, Project Work, World Work) are not backed up and are purged on a regular basis according to the timeframes listed above.

[1] In addition, there is a quota/limit of 2,000 files on this directory.

[2] Permissions on Member Work directories can be controlled to an extent by project members. By default, only the project member has any accesses, but accesses can be granted to other project members by setting group permissions accordingly on the Member Work directory. The parent directory of the Member Work directory prevents accesses by “UNIX-others” and cannot be changed (security measures).

[3] In addition, there is a quota/limit of 100,000 files on this directory.

[4] Retention is not applicable as files will follow purge cycle.

Data Retention Overview

By default, there is no lifetime retention for any data on OLCF resources. The OLCF specifies a limited post-deactivation timeframe during which user and project data will be retained. When the retention timeframe expires, the OLCF retains the right to delete data. If you have data retention needs outside of the default policy, please notify the OLCF.

User Data Retention

The user data retention policy exists to reclaim storage space after a user account is deactivated, e.g., after the user’s involvement on all OLCF projects concludes. By default, the OLCF will retain data in user-centric storage areas only for a designated amount of time after the user’s account is deactivated. During this time, a user can request a temporary user account extension for data access. See the section “Data Retention, Purge, & Quota Summary” for details on retention timeframes for each user-centric storage area.

Project Data Retention

The project data retention policy exists to reclaim storage space after a project ends. By default, the OLCF will retain data in project-centric storage areas only for a designated amount of time after the project end date. During this time, a project member can request a temporary user account extension for data access. See the section “Data Retention, Purge, & Quota Summary” for details on purge and retention timeframes for each project-centric storage area.

Sensitive Project Data Retention

For sensitive projects only, all data related to the project must be purged from all OLCF computing resources within 30 days of the project’s end or termination date.

Data Purges

Data purge mechanisms are enabled on some OLCF file system directories in order to maintain sufficient disk space availability for job execution. Files in these scratch areas are automatically purged on a regular purge timeframe. If a file system with an active purge policy is nearing capacity, the OLCF may contact you to request that you reduce the size of a directory within that file system, even if the purge timeframe has not been exceeded. See the section “Data Retention, Purge, & Quota Summary” for details on purge timeframes for each storage area, if applicable.

Storage Space Quotas

Each user-centric and project-centric storage area has an associated quota, which could be a hard (systematically-enforceable) quota or a soft (policy-enforceable) quota. Storage usage will be monitored continually. When a user or project exceeds a soft quota for a storage area, the user or project PI will be contacted and will be asked if at all possible to purge data from the offending area. See the section “Data Retention, Purge, & Quota Summary” for details on quotas for each storage area.

Data Prohibitions & Safeguards
Prohibited Data

The OLCF computer systems are operated as research systems and only contain data related to scientific research and do not contain personally identifiable information (data that falls under the Privacy Act of 1974 5U.S.C. 552a). Use of OLCF resources to store, manipulate, or remotely access any national security information is strictly prohibited. This includes, but is not limited to: classified information, unclassified controlled nuclear information (UCNI), naval nuclear propulsion information (NNPI), the design or development of nuclear, biological, or chemical weapons or any weapons of mass destruction. Authors/generators/owners of information are responsible for its correct categorization as sensitive or non-sensitive. Owners of sensitive information are responsible for its secure handling, transmission, processing, storage, and disposal on OLCF systems. Principal investigators, users, or project delegates that use OLCF resources, or are responsible for overseeing projects that use OLCF resources, are strictly responsible for knowing whether their project generates any of these prohibited data types or information that falls under Export Control. For questions, contact help@olcf.ornl.gov.

Unauthorized Data Modification

Users are prohibited from taking unauthorized actions to intentionally modify or delete information or programs.

Data Confidentiality, Integrity, & Availability

The OLCF systems provide protections to maintain the confidentiality, integrity, and availability of user data. Measures include: the availability of file permissions, archival systems with access control lists, and parity/CRC checks on data paths/files. It is the user’s responsibility to set access controls appropriately for data. In the event of system failure or malicious actions, the OLCF makes no guarantee against loss of data nor makes a guarantee that a user’s data could not be potentially accessed, changed, or deleted by another individual. It is the user’s responsibility to insure the appropriate level of backup and integrity checks on critical data and programs.

Administrator Access to Data

OLCF resources are federal computer systems, and as such, users should have no explicit or implicit expectation of privacy. OLCF employees and authorized vendor personnel with “root” privileges have access to all data on OLCF systems. Such employees can also login to OLCF systems as other users.

As a general rule, OLCF employees will not discuss your data with any unauthorized entities nor grant access to data files to any person other than the UNIX “owner” of the data file, except in the following situations:

  • When the owner of the data requests a change of ownership for any reason, e.g., the owner is leaving the project and grants the PI ownership of the data.
  • In situations of suspected abuse/misuse computational resources, criminal activity, or cyber-security violations.

Note that the above applies even to project PIs. In general, the OLCF will not overwrite existing UNIX permissions on data files owned by project members for the purpose of granting access to the project PI. Project PIs should work closely with project members throughout the duration of the project to ensure UNIX permissions are set appropriately.

Software
Software Licensing

All software used on OLCF computers must be appropriately acquired and used according to the appropriate software license agreement. Possession, use, or transmission of illegally obtained software is prohibited. Likewise, users shall not copy, store, or transfer copyrighted software, except as permitted by the owner of the copyright. Only export-controlled codes approved by the Export Control Office may be run by parties with sensitive data agreements.

Malicious Software

Users must not intentionally introduce or use malicious software, including but not limited to, computer viruses, Trojan horses, or computer worms.

Reconstruction of Information or Software

Users are not permitted to reconstruct information or software for which they are not authorized. This includes but is not limited to any reverse engineering of copyrighted software or firmware present on OLCF computing resources.