titan

Up since 11/8/17 02:45 pm

eos

Up since 11/14/17 11:20 pm

rhea

Up since 10/17/17 05:40 pm

hpss

Up since 11/20/17 09:15 am

atlas1

Up since 11/15/17 07:25 am

atlas2

Up since 11/27/17 10:45 am
OLCF User Assistance Center

Can't find the information you need below? Need advice from a real person? We're here to help.

OLCF support consultants are available to respond to your emails and phone calls from 9:00 a.m. to 5:00 p.m. EST, Monday through Friday, exclusive of holidays. Emails received outside of regular support hours will be addressed the next business day.

Data Transfer Between OLCF Systems

The OLCF provides a shared-storage environment, so transferring data between our machines is largely unnecessary. However, we provide tools both to move large amounts of data between scratch and archival storage and from one scratch area to another.

Intra-Filesystems Transfers

The following utilities can be used to transfer data between partitions of the filesystem.

Utility Amount of Data to Transfer Where to run utility Notes
cp < 500 GB command line / batch script Useful when transferring small amounts of data, available from Titan and the DTNs
bbcp < 500 GB DTN command line or batch script Multi-streaming ability can make bbcp a faster option than cp, should be executed from DTNs only
fcp > 500 GB batch script Faster than cp and bbcp for directory trees, can preserve lustre striping, only available from the batch scheduled DTNs

For large transfers, transfers to the High Performance Storage System (discussed below), and when running fcp or bbcp, the DTNs should be used to prevent overloading the compute system’s login nodes.

To help reduce load on the compute systems’ login resources, please use the DTNs when using fcp or bbcp, or transferring more than ~500GB.

Transfers to the High Performance Storage System

Since all areas of the shared luster filesystem are periodically swept, moving data from lustre to archival storage on High performance storage system is a necessary part of most users’ work flow.  The commands hsi and htar provide users with easy-to-use interfaces to their User Archive and Project Archive spaces on the OLCF’s HPSS-based archival storage system.

The Transferring Data with HSI and HTAR page contains examples of hsi and htar and sample workflows.