Publishing Notice Prior to publishing work performed on Summit, please contact help@olcf.ornl.gov.

Storage Overview

OLCF users have many options for data storage. Each user has multiple user-affiliated storage spaces, and each project has multiple project-affiliated storage spaces where data can be shared for collaboration. Below we give an overview and explain where each storage area is mounted.

AlpineTDS IBM Spectrum Scale Filesystem

Currently Summit mounts a POSIX-based IBM Spectrum Scale parallel filesystem called AlpineTDS. It is the initial test and development system that will be followed by the full production Spectrum Scale filesystem called Alpine. Early Summit users are expected to do their production I/O using AlpineTDS until the full Alpine is deployed. AlpineTDS has an approximate capacity of 3 PB and an approximate maximum speed of 30 GB/s. AlpineTDS will be removed from production and reformatted after the Alpine filesystem is available for production. Alpine’s maximum capacity will be 250 PB. The maximum performance of the final production system will be about 2.5 TB/s for sequential I/O and 2.2 TB/s for random I/O.

For best performance on the IBM Spectrum Scale filesystem, use large pagealigned I/O and asynchronous reads and writes. The filesystem blocksize is 16MB, The minimum fragment size is 16K so when a file under 16K is stored, it will still use 16K of the disk. Files writes of 16 MB or larger, will achieve better performance. All files are striped across LUNs which are distributed across all IO servers.

Other OLCF Storage Systems

The High Performance Storage System (HPSS) at the OLCF provides longer-term storage for the large amounts of data created on the OLCF compute systems. The HPSS is accessible from all OLCF Filesystems through utilities called HSI and HTAR. For more information on using HSI or HTAR, see the HPSS Best Practices documentation.

OLCF also has a Network File System, referred to as NFS, and Lustre filesystems called Atlas. Summit does not mount Lustre. However, during the early use of Summit, users may need to use Lustre in a multi-stage process with HPSS for larger data transfer with AlpineTDS. To learn more about this please see Data Transfer and Summit section below.

The following shows the availability of each of the filesystems on primary OLCF clusters and supercomputers.

Area Summit Titan Data Transfer Nodes Rhea Eos
Atlas Lustre Filesystem yes yes yes yes
AlpineTDS Spectrum Scale Filesystem yes
NFS Network Filesystem yes yes yes yes yes
HPSS HSI/Htar HSI/Htar HSI/Htar HSI/Htar HSI/Htar


Storage Areas

The storage area to use in any given situation depends upon the activity you wish to carry out.
Each user has a User Home area on a Network File System (NFS) and a User Archive area on the archival High Performance Storage System (HPSS). These user storage areas are intended to house user-specific files.

Each project has a Project Home area on NFS, multiple Project Work areas on Lustre and Spectrum Scale, and a Project Archive area on HPSS. These project storage areas are intended to house project-centric files.

We have defined several areas as listed below by function:

  • User Home: Long-term data for routine access that is unrelated to a project.
  • User Archive: Long-term data for archival access that is unrelated to a project.
  • Project Home: Long-term project data for routine access that’s shared with other project members.
  • Member Work: Short-term user data for fast, batch-job access that is not shared with other project members. There are versions of this on both the Atlas Lustre filesystem and the AlpineTDS Spectrum Scale filesystem.
  • Project Work: Short-term project data for fast, batch-job access that’s shared with other project members. There are versions of this on both the Atlas Lustre filesystem and the AlpineTDS Spectrum Scale filesystem.
  • World Work: Short-term project data for fast, batch-job access that’s shared with OLCF users outside your project. There are versions of this on both the Atlas Lustre filesystem and the AlpineTDS Spectrum Scale filesystem.
  • Project Archive: Long-term project data for archival access that’s shared with other project members.

Guidelines
A brief description of each area and basic guidelines to follow are provided in the table below:

Name Path Type Permissions Backups Purged Quota
User Home $HOME NFS User Set yes no 50GB
User Archive /home/$USER HPSS User Set no no 2TB
Project Home /ccs/proj/[projid] NFS 770 yes no 50GB

Member Work
/lustre/atlas/scratch/[userid]/[projid] Lustre 700 no 14 days 10 TB
/gpfs/alpinetds/scratch/[userid]/[projid] Spectrum Scale 700 no TBD TBD

Project Work
/lustre/atlas/proj-shared/[projid] Lustre 770 no 90 days 100 TB
/gpfs/alpinetds/proj-shared/[projid] Spectrum Scale 770 no TBD TBD

World Work
/lustre/atlas/world-shared/[projid] Lustre 775 no 90 days 10TB
/gpfs/alpinetds/world-shared/[projid] Spectrum Scale 775 no TBD TBD
Project Archive /proj/[projid] HPSS 770 no no 100TB



On Titan, Rhea, and Eos, paths to the various project-centric work storage areas are simplified by the use of environment variables that point to the proper directory on a per-user basis:

  • Member Work Directory: $MEMBERWORK/[projid]
  • Project Work Directory: $PROJWORK/[projid]
  • World Work Directory: $WORLDWORK/[projid]
  • These environment variables are not set on Summit or the data transfer nodes.

    Purge

    To keep the Lustre and Spectrum Scale filesytems exceptionally performant, untouched files in the project and user areas are purged at the intervals shown in the table above. Please make sure that valuable data is moved off of these systems regularly. See HPSS Best Practices for information about using the HSI and HTAR utilities to archive data on HPSS.

    Backups for Files on NFS

    Online backups are performed at regular intervals for your files in project home and user home. Hourly backups for the past 24 hours, daily backups for the last 7 days, and 1 weekly backup are available. The backup directories are named hourly.*, daily.* , and weekly.* where * is the date/time stamp of the backup. For example, hourly.2016-12-01-0905 is an hourly backup made on December 1, 2016 at 9:05 AM.

    The backups are accessed via the .snapshot subdirectory. You may list your available hourly/daily/weekly backups by doing “ls .snapshot”. The .snapshot feature is available in any subdirectory of your home directory and will show the online backup of that subdirectory. In other words, you don’t have to start at /ccs/home/$USER and navigate the full directory structure; if you’re in a /ccs/home subdirectory several “levels” deep, an “ls .snapshot” will access the available backups of that subdirectory.

    To retrieve a backup, simply copy it into your desired destination with the cp command.

    Retention

    At the completion of a project or at the end of a member’s association with the project, data will be retained for 90 days, except in areas that are purged, in that case the data will be retained according the purge policy.

    A more detailed description of each storage area is given below.

    User-Centric Data Storage

    The following table summarizes user-centric storage areas available on OLCF resources and lists relevant polices.

    User-Centric Storage Areas
    Area Path Type Permissions Quota Backups Purged Retention
    User Home $HOME NFS User-controlled 10 GB Yes No 90 days
    User Archive /home/$USER HPSS User-controlled 2 TB [1] No No 90 days
    [1] In addition, there is a quota/limit of 2,000 files on this directory.

    User Home Directories (NFS)

    The environment variable $HOME will always point to your current home directory. It is recommended, where possible, that you use this variable to reference your home directory. In cases in which using $HOME is not feasible, it is recommended that you use /ccs/home/$USER.

    Users should note that since this is an NFS-mounted filesystem, its performance will not be as high as other filesystems.

    User Home Quotas

    Quotas are enforced on user home directories. To request an increased quota, contact the OLCF User Assistance Center. To view your current quota and usage, use the quota command:
    $ quota -Qs
    Disk quotas for user usrid (uid 12345):
         Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
    nccsfiler1a.ccs.ornl.gov:/vol/home
                      4858M   5000M   5000M           29379   4295m   4295m
    

    User Home Permissions

    The default permissions for user home directories are 0750 (full access to the user, read and execute for the group). Users have the ability to change permissions on their home directories, although it is recommended that permissions be set to as restrictive as possible (without interfering with your work).

    User Website Directory

    Users interested in sharing files publicly via the World Wide Web can request a user website directory be created for their account. User website directories (~/www) have a 5GB storage quota and allow access to files at http://users.nccs.gov/~user (where user is your userid). If you are interested in having a user website directory created, please contact the User Assistance Center at help@olcf.ornl.gov.

    User Archive Directories (HPSS)

    The High Performance Storage System (HPSS) at the OLCF provides longer-term storage for the large amounts of data created on the OLCF compute systems. The mass storage facility consists of tape and disk storage components, servers, and the HPSS software. After data is uploaded, it persists on disk for some period of time. The length of its life on disk is determined by how full the disk caches become. When data is migrated to tape, it is done so in a first-in, first-out fashion.

    User archive areas on HPSS are intended for storage of data not immediately needed in either User Home directories (NFS) or User Work directories (Lustre®). User Archive directories should not be used to store project-related data. Rather, Project Archive directories should be used for project data.

    User archive directories are located at /home/$USER.

    User Archive Access

    Each OLCF user receives an HPSS account automatically. Users can transfer data to HPSS from any OLCF system using the HSI or HTAR utilities. For more information on using HSI or HTAR, see the HPSS Best Practices section.

    User Archive Accounting

    Each file and directory on HPSS is associated with an HPSS storage allocation. For information on HPSS storage allocations, please visit the HPSS Archive Accounting section.

    For information on usage and best practices for HPSS, please see the HPSS Best Practices documentation.


    Project-Centric Data Storage

    Project directories provide members of a project with a common place to store code, data, and other files related to their project.

    Project Home Directories (NFS)

    Name Path Type Permissions Backups Purged Quota
    Project Home /ccs/proj/[projid] NFS 770 yes no 50GB

    Projects are provided with a Project Home storage area in the NFS-mounted filesystem. This area is intended for storage of data, code, and other files that are of interest to all members of a project. Since Project Home is an NFS-mounted filesystem, its performance will not be as high as other filesystems.

    Project Home Path

    Project Home area is accessible at /ccs/proj/abc123 (where abc123 is your project ID).

    Project Home Quotas

    To check your project’s current usage, run df -h /ccs/proj/abc123 (where abc123 is your project ID). Quotas are enforced on project home directories. The current limit is shown in the table above.

    Project Home Permissions

    The default permissions for project home directories are 0770 (full access to the user and group). The directory is owned by root and the group includes the project’s group members. All members of a project should also be members of that group-specific project. For example, all members of project “ABC123” should be members of the “abc123” UNIX group.

    Three Project Work Areas to Facilitate Collaboration

    To facilitate collaboration among researchers, the OLCF provides (3) distinct types of project-centric work storage areas: Member Work directories, Project Work directories, and World Work directories. Each directory should be used for storing files generated by computationally-intensive HPC jobs related to a project.

    Name Path Type Permissions Backups Purged Quota

    Member Work
    /lustre/atlas/scratch/[projid] Lustre 700 no 14 days 10 TB
    /gpfs/alpinetds/scratch/[projid] Spectrum Scale 700 no 90 days TBD

    Project Work
    /lustre/atlas/proj-shared/[projid] Lustre 770 no 90 days 100 TB
    /gpfs/alpinetds/proj-shared/[projid] Spectrum Scale 770 no 90 days TBD

    World Work
    /lustre/atlas/world-shared/[projid] Lustre 775 no 90 days 10TB
    /gpfs/alpinetds/world-shared/[projid] Spectrum Scale 775 no 90 days TBD


    The difference between the three lies in the accessibility of the data to project members and to researchers outside of the project. Member Work directories are accessible only by an individual project member by default. Project Work directories are accessible by all project members. World Work directories are readable by any user on the system.

    Permissions

    UNIX Permissions on each project-centric work storage area differ according to the area’s intended collaborative use. Under this setup, the process of sharing data with other researchers amounts to simply ensuring that the data resides in the proper work directory.

    • Member Work Directory: 700
    • Project Work Directory: 770
    • World Work Directory: 775

    For example, if you have data that must be restricted only to yourself, keep them in your Member Work directory for that project (and leave the default permissions unchanged). If you have data that you intend to share with researchers within your project, keep them in the project’s Project Work directory. If you have data that you intend to share with researchers outside of a project, keep them in the project’s World Work directory.

    Backups

    Member Work, Project Work, and World Work directories are not backed up. Project members are responsible for backing up these files, either to Project Archive areas (HPSS) or to an off-site location.

    Project Archive Directories

    Name Path Type Permissions Backups Purged Quota
    Project Archive /proj/[projid] HPSS 770 no no 100TB

    Projects are also allocated project-specific archival space on the High Performance Storage System (HPSS). The default quota is shown on the table above. If a higher quota is needed, contact the User Assistance Center.

    The Project Archive space on HPSS is intended for storage of data not immediately needed in either Project Home (NFS) areas nor Project Work (AlpineTDS) areas, and to serve as a location to store backup copies of project-related files.

    Project Archive Path

    The project archive directories are located at /proj/pjt000 (where pjt000 is your Project ID).

    Project Archive Access

    Project Archive directories may only be accessed via utilities called HSI and HTAR. For more information on using HSI or HTAR, see the HPSS Best Practices section.

    Data Transfer and Summit

    Early Summit users will have access to a 3 PB GPFS file system call AlpineTDS. Because this file system is shared by all users, and due to its limited size, it is important to transfer your data off of AlpineTDS whenever possible. Another reason it is important to regularly transfer your data off of AlpineTDS is that the system is still under development, so it is possible your data could be lost without warning.

    AlpineTDS will not be mounted on OLCF’s data transfer nodes, so transfers will be staged on the Summit login nodes. However, when the full Alpine file system goes into production, it will be mounted on the data transfer nodes as well as Summit, and the data transfer process will become enabled by tools like Globus.

    AlpineTDS will be removed and reformatted after the full Alpine file system is available for production. To avoid loss of data, and to make the transition to Alpine easier, users should archive valuable project data created on AlpineTDS in HPSS. Please see HPSS Best Practices for information about using the HSI and HTAR utilities to archive data on HPSS.

    The following sections will outline the current options for transferring data to and from AlpineTDS.

    Local Transfers

    For local data transfers (e.g. from AlpineTDS to Atlas), users have two options depending on the size of the data. For small data transfers, users can first copy data (via the cp command) from AlpineTDS to the NFS storage area (i.e. home /ccs/home/userid/ or project space /ccs/proj/), and then from their NFS directory to Atlas (/lustre/atlas). Because the NFS storage areas have a 50 GB quota, such transfers must be less than 50 GB.

    If larger data must be moved between AlpineTDS and Atlas, it can be passed through HPSS. For example, if your data is initially located in /lustre/atlas/proj-shared/[projid] and you want to move it to /gpfs/alpinetds/scratch/[projid]:

    Log in to titan.ccs.ornl.gov or dtn.css.ornl.gov

    cd  /lustre/atlas/proj-shared/[projid]
    htar -cvf mylargefiles.tar mylargefiles
    

    Login to summit.olcf.ornl.gov

    cd /gpfs/alpinetds/scratch/[projid]
    htar -xvf mylargefiles.tar
    


    Below is a screencast showing the process of transferring data from AlpineTDS to Atlas.

    Remote Transfers with AlpineTDS

    scp and rsync are available on Summit for small remote transfers.

    For larger remote transfers with AlpineTDS, we recommend staging the data through Atlas and using Globus to do the remote transfer. In this case, HPSS can be used as a pass-through between Atlas and AlpineTDS as mentioned above.

    Please see the Titan User Guide’s Remote Transfers section for additional options (and more detailed information) about how to transfer data between Atlas and systems located outside OLCF.

    HPSS Best Practices

    Currently HSI and HTAR are offered for archiving data into HPSS or retrieving data from the HPSS archive.

    For optimal transfer performance we recommend sending file of 768 GB or larger to HPSS. The minimum file size that we recommend sending is 512 MB. HPSS will handle files between 0K and 512 MB, but write and read performance will be negatively affected. For files smaller than 512 MB we recommend bundling them with HTAR to achieve an archive file of at least 512 MB.

    When retrieving data from a tar archive larger than 1 TB, we recommend that you pull only the files that you need rather than the full archive. Examples of this will be give in the htar section below.

    Using HSI

    Issuing the command hsi will start HSI in interactive mode. Alternatively, you can use:

      hsi [options] command(s)
    

    …to execute a set of HSI commands and then return.

    To list you files on the HPSS, you might use:

      hsi ls
    

    hsi commands are similar to ftp commands. For example, hsi get and hsi put are used to retrieve and store individual files, and hsi mget and hsi mput can be used to retrieve multiple files.

    To send a file to HPSS, you might use:

      hsi put a.out
    

    To put a file in a pre-existing directory on hpss:

      hsi “cd MyHpssDir; put a.out”
    

    To retrieve one, you might use:

      hsi get /proj/projectid/a.out
    

    Here is a list of commonly used hsi commands.

    Command Function
    cd Change current directory
    get, mget Copy one or more HPSS-resident files to local files
    cget Conditional get – get the file only if it doesn’t already exist
    cp Copy a file within HPSS
    rm mdelete Remove one or more files from HPSS
    ls List a directory
    put, mput Copy one or more local files to HPSS
    cput Conditional put – copy the file into HPSS unless it is already there
    pwd Print current directory
    mv Rename an HPSS file
    mkdir Create an HPSS directory
    rmdir Delete an HPSS directory

     

    Additional HSI Documentation

    There is interactive documentation on the hsi command available by running:

      hsi help
    

    Additionally, documentation can be found at the Gleicher Enterprises website, including an HSI Reference Manual and man pages for HSI.

    Using HTAR

    The htar command provides an interface very similar to the traditional tar command found on UNIX systems. It is used as a command-line interface. The basic syntax of htar is:

    htar -{c|K|t|x|X} -f tarfile [directories] [files]

    As with the standard Unix tar utility the -c, -x, and -t options, respectively, function to create, extract, and list tar archive files. The -K option verifies an existing tarfile in HPSS and the -X option can be used to re-create the index file for an existing archive.

    For example, to store all files in the directory dir1 to a file named allfiles.tar on HPSS, use the command:

      htar -cvf allfiles.tar dir1/*
    

    To retrieve these files:

      htar -xvf allfiles.tar 
    

    htar will overwrite files of the same name in the target directory.

    When possible, extract only the files you need from large archives.

    To display the names of the files in the project1.tar archive file within the HPSS home directory:

      htar -vtf project1.tar
    

    To extract only one file, executable.out, from the project1 directory in the Archive file called project1.tar:

      htar -xm -f project1.tar project1/ executable.out 
    

    To extract all files from the project1/src directory in the archive file called project1.tar, and use the time of extraction as the modification time, use the following command:

      htar -xm -f project1.tar project1/src
    
    HTAR Limitations

    The htar utility has several limitations.

    Apending data

    You cannot add or append files to an existing archive.

    File Path Length

    File path names within an htar archive of the form prefix/name are limited to 154 characters for the prefix and 99 characters for the file name. Link names cannot exceed 99 characters.

    Size

    There are limits to the size and number of files that can be placed in an HTAR archive.

    Individual File Size Maximum 68GB, due to POSIX limit
    Maximum Number of Files per Archive 1 million

     

    For example, when attempting to HTAR a directory with one member file larger that 64GB, the following error message will appear:

    
    [titan-ext1]$htar -cvf hpss_test.tar hpss_test/
    
    INFO: File too large for htar to handle: hpss_test/75GB.dat (75161927680 bytes)
    ERROR: 1 oversize member files found - please correct and retry
    ERROR: [FATAL] error(s) generating filename list 
    HTAR: HTAR FAILED
    
    Additional HTAR Documentation


    The HTAR user’s guide can be found at the Gleicher Enterprises website Gleicher Enterprises website, including the HTAR man page.