2019 OLCF System Decommissions and Notable Changes

As the OLCF begins preparations for Frontier, you should be aware of plans that will impact Titan, Eos, Rhea, and the Atlas filesystem in 2019.

Please pay attention to the following key dates as you plan your science campaigns and data migration for the remainder of the year:

Date Event
March 01 The OLCF will no longer accept Director’s Discretionary proposals for the Titan and Eos computational resources
March 12 Rhea’s OS upgraded to RHEL7
June 25 Rhea will transition from mounting the Atlas Lustre filesystem to mounting the Alpine GPFS filesystem
June 30 Last day to submit jobs to Titan and Eos
August 01 Atlas becomes read-only. Please do not wait, begin transferring your Lustre data now.
August 01 Titan and Eos will be decommissioned
August 15 Atlas decommissioned; ALL REMAINING DATA on ATLAS will be PERMANENTLY DELETED
August 27 DTNs will transition from the Moab scheduler to the Slurm scheduler (tentative)
September 03 Rhea will transition from the Moab scheduler to the Slurm scheduler (tentative)

Event Details

March 01, 2019 —- DD projects no longer accepted for Titan and Eos

Due to the limited remaining compute time on Titan and Eos, the OLCF will no longer accept Director’s Discretion projects as of March 01, 2019.

March 12, 2019 —- Rhea RHEL7 upgrade

Rhea will be upgraded to RHEL7 on March 12. Please note the following:

Recompile recommended
Due to the OS upgrade and other software changes, we recommend recompiling prior to running on the upgraded system.
FreeNX decommissioning
FreeNX will be no longer be available from Rhea following the upgrade. VNC will remain available from Rhea’s compute nodes.
More information on using VNC on OLCF systems, can be found here .
Environment Modules transition to Lmod
The environment module management system will be updated to Lmod. Lmod is a Lua-based module system for dynamically altering shell environments. Typical use of Lmod is very similar to that of interacting with modulefiles on other OLCF systems. The interface to Lmod is still provided by the module command. However; there are a number of changes, including hierarchical structure, that help interaction but should be noted.
More information on using Lmod on Rhea can be found here.
Default software version changes
Default versions of a number of software packages, including compilers, OpenMPI, and CUDA, will be changing during the upgrade. Due to the OS upgrade and software changes, users should rebuild prior to running on the upgraded system.
The list of new defaults can be found here. If you do not see a required package on the list, please notify help@olcf.ornl.gov
Python runtime management
The python runtime will change to python/X.Y.Z-anacondaX-5.3.0. python/x.y.z-anacondax-5.3.0 should be used instead of python/x.y.z for a complete environment. Conda environments can be used for packages not already installed on system.

June 25, 2019 —- Rhea transition from Atlas to GPFS

On June 25, Rhea will unmount the Lustre scratch filesystem, Atlas, and mount the new GPFS scratch filesystem, Alpine. You need to transfer your data from Atlas to Alpine in order to use them from Rhea until Atlas is decommissioned.
Following this change, the Atlas filesystem will no longer be accessible from Rhea. More information on this change will be posted here as we work through the details. More information on transferring data from Atlas to Alpine can be found here.

June 30, 2019 —- Last day to submit jobs on Titan and Eos

The last day to submit batch jobs on Titan and Eos will be June 30, 2019.

August 01, 2019 —- Atlas becomes read-only

In preparation for Atlas’ decommission the Lustre scratch filesystem, Atlas, will become read-only from all OLCF systems on August 01. Please do not wait, begin transferring your Lustre data now.

Moving data off-site
Globus is the suggested tool to move data off-site
Standard tools such as rsync and scp can also be used, but may be slower and require more manual intervention than Globus
Copying data directly from Atlas (Lustre) to Alpine (GPFS)
Globus is the suggested tool to transfer needed data from Atlas to Alpine. Globus should be used when transfer large amounts of data.
Standard tools such as rsync and cp can also be used. The DTN mounts both filesystems and should be used when transferring with rsync and cp tools. These methods should not be used to transfer large amounts of data.
Copying data to the HPSS archive system
The hsi and htar utilities can be used to to transfer data from the Atlas filesystem to the HPSS. The tools can also be used to transfer data from the HPSS to the Alpine filesystem.
Globus is also now available to transfer data directly to the HPSS

More information on transferring data from Atlas to Alpine can be found here.

August 01, 2019 —- Titan and Eos will be decommissioned

Titan and Eos will be decommissioned on August 01. On this date logins to Titan and Eos will no longer be allowed.

August 15, 2019 —- All remaining data on Atlas will be permanently deleted

Following August 15, data remaining on the Lustre filesystem, Atlas, will no longer be accessible and will be permanently deleted . Following this date, the OLCF will no longer be able to retrieve data remaining on Atlas. Due to the large amount of data on the filesystems, we strongly urge you to start transferring your data now, and do not wait until later in the year. More information on transferring data from Atlas to Alpine can be found here.

August 27, 2019 —- DTNs will transition to the Slurm batch scheduler

The batch scheduler on the DTNs will migrate from the Moab scheduler to the Slurm batch scheduler. Instructions on how to use SLURM scheduler will be provided soon.

September 03, 2019 —- Rhea will transition to the Slurm batch scheduler

The batch scheduler on Rhea will migrate from the Moab scheduler to the Slurm batch scheduler. Instructions on how to use SLURM scheduler will be provided soon.

 


Atlas to Alpine Data Transfer

Below are instructions on how to copy your files from Atlas to Alpine. We are going to mention three approaches. The first one is Globus, the second one HPSS, and the third one are the DTN servers using your own scripts. If you would like to backup your files or you don’t want your files to be transferred through the network outside the ORNL, then use HPSS. However, in many cases Globus is faster in transferring files from Atlas to Alpine.

Globus

Initially, you can see a video with instructions, and below screenshots for step by step instructions for using Globus.

 

Follow the steps:

  • Visit www.globus.org and login

  • Then select your organization if it is in the list or you need to create a Globus account

 

  • Search for the endpoint OLCF DTN

 

  • Declare path

  • Open a second panel to declare where your files would like to be transferred, select if you want an encrypt transfer, select your file/folder and click start

 

 

 

  • Then an activity report will appear and you can click on it to see the status. When the transfer is finished or failed, you will receive an email

When the transfer starts, you could close your browser, if you transfer a lot of data, it will take some time. You will receive email with the outcome. For any issue contact help@olcf.ornl.gov

HPSS

Using Globus

The OLCF users have access to a new functionality, using Globus to transfer files to HPSS through the endpoint “OLCF HPSS“. Globus has restriction of 8 active transfers across all the users. Each user has a limit of 3 active transfers, so it is required to transfer a lot of data on each transfer than less data across many transfers. If a folder is constituted with mixed files including thousands of small files (less than 1MB each one), it would be better to tar the small files. Otherwise, if the files are larger, Globus will handle them.

To transfer the files, follow the steps which some are identical with the instructions above regarding transfer files from Atlas to Alpine:

  • Visit www.globus.org and login

  • Choose organization:

  • Use your credentials to login

  • Search for the endpoint OLCF DTN

 

  • Declare path

  • Open a second panel to declare where your files would like to be transferred

  • Declare the new endpoint called OLCF HPSS and use the appropriate path for HPSS

  • Select if you want an encrypt transfer, select your file/folder and click start

 

  • Then an activity report will appear and you can click on it to see the status. When the transfer is finished or failed, you will receive an email

  • When you click on the activity of the corresponding transfer, you can see the status or the results when it is finished

Using the tools

Currently, HSI and HTAR are offered for archiving data into HPSS or retrieving data from the HPSS archive. For better performance connect to DTN and execute the hsi/htar commands from there.

For optimal transfer performance, we recommend sending a file of 768 GB or larger to HPSS. The minimum file size that we recommend sending is 512 MB. HPSS will handle files between 0K and 512 MB, but write and read performance will be negatively affected. For files smaller than 512 MB we recommend bundling them with HTAR to achieve an archive file of at least 512 MB.

When retrieving data from a tar archive larger than 1 TB, we recommend that you pull only the files that you need rather than the full archive. Examples of this will be given in the htar section below.

Using HSI

Issuing the command hsi will start HSI in interactive mode. Alternatively, you can use:

  hsi [options] command(s)

…to execute a set of HSI commands and then return.

To list you files on the HPSS, you might use:

  hsi ls

HSI get and put commands also retrieve and store multiple files; mget and mput were
added for compatibility with ftp. In addition, both get and put can use csh-style wildcard patterns
and support recursion, so that entire directory trees can be retrieved or stored with a simple command
such as:

  hsi get -R hpss_directory

or

  hsi put -R local_directory

To send a file to HPSS, you might use:

  hsi put a.out

To put a file in a pre-existing directory on hpss:

  hsi “cd MyHpssDir; put a.out”

To retrieve one, you might use:

  hsi get /proj/projectid/a.out

Here is a list of commonly used hsi commands.

Command Function
cd Change current directory
get, mget Copy one or more HPSS-resident files to local files
cget Conditional get – get the file only if it doesn’t already exist
cp Copy a file within HPSS
rm mdelete Remove one or more files from HPSS
ls List a directory
put, mput Copy one or more local files to HPSS
cput Conditional put – copy the file into HPSS unless it is already there
pwd Print current directory
mv Rename an HPSS file
mkdir Create an HPSS directory
rmdir Delete an HPSS directory
Scenario 1:

Transfer file test.tar from the path /lustre/atlas/path/ to /proj/projid while the current directory is the /lustre/atlas/path/

  hsi "cd /proj/projid; put test.tar"
Scenario 2:

Transfer file test.tar from the path /lustre/atlas/path/ to /proj/projid while the current working directory is not the /lustre/atlas/path/

  hsi "put  /lustre/atlas/path/test.tar : /proj/projid/test.tar"
Additional HSI Documentation

There is interactive documentation on the hsi command available by running:

  hsi help

Additionally, documentation can be found at the Gleicher Enterprises website, including an HSI Reference Manual and man pages for HSI.

Using HTAR

The htar command provides an interface very similar to the traditional tar command found on UNIX systems. It is used as a command-line interface. The basic syntax of htar is:

htar -{c|K|t|x|X} -f tarfile [directories] [files]

As with the standard Unix tar utility the -c, -x, and -t options, respectively, function to create, extract, and list tar archive files. The -K option verifies an existing tarfile in HPSS and the -X option can be used to re-create the index file for an existing archive.

For example, to store all files in the directory dir1 to a file named allfiles.tar on HPSS, use the command:

  htar -cvf allfiles.tar dir1/*

To retrieve these files:

  htar -xvf allfiles.tar 

htar will overwrite files of the same name in the target directory.

When possible, extract only the files you need from large archives.

To display the names of the files in the project1.tar archive file within the HPSS home directory:

  htar -vtf project1.tar

To extract only one file, executable.out, from the project1 directory in the Archive file called project1.tar:

  htar -xm -f project1.tar project1/ executable.out 

To extract all files from the project1/src directory in the archive file called project1.tar, and use the time of extraction as the modification time, use the following command:

  htar -xm -f project1.tar project1/src
HTAR Limitations

The htar utility has several limitations.

Apending data

You cannot add or append files to an existing archive.

File Path Length

File path names within an htar archive of the form prefix/name are limited to 154 characters for the prefix and 99 characters for the file name. Link names cannot exceed 99 characters.

Size

There are limits to the size and number of files that can be placed in an HTAR archive.

Individual File Size Maximum 68GB, due to POSIX limit
Maximum Number of Files per Archive 1 million

For example, when attempting to HTAR a directory with one member file larger that 64GB, the following error message will appear:

[titan-ext1]$htar -cvf hpss_test.tar hpss_test/

INFO: File too large for htar to handle: hpss_test/75GB.dat (75161927680 bytes)
ERROR: 1 oversize member files found - please correct and retry
ERROR: [FATAL] error(s) generating filename list 
HTAR: HTAR FAILED
Scenario 3:

The command htar fails because of the limitations of the tool, so we tar the folder and create a pipeline with hsi to created a tar file called test.tar in /proj/projid/

  tar cf - directory_to_compress | hsi "cd /proj/projid; put - : test.tar"

You can use the previously mentioned command htar -xvf to retrieve the file from Alpine, however, consider to extract only the necessary files.

Additional HTAR Documentation

The HTAR user’s guide can be found at the Gleicher Enterprises website Gleicher Enterprises website, including the HTAR man page.

DTN

You can use Data Transfer Nodes (DTN) to transfer files with scp and rsync, however, probably the other proposed approaches could be faster if you have not prepared and optimized script for efficient performance.

  ssh your_username@dtn.ccs.ornl.gov

Then you have access to /lustre/atlas and /gpfs/alpine in order to use a script to transfer your files.