Data Transfer Between OLCF Systems
Print this article
The OLCF provides a shared-storage environment, so transferring data between our machines is largely unnecessary. However, we provide tools both to move large amounts of data between scratch and archival storage and from one scratch area to another.
The following utilities can be used to transfer data between partitions of the filesystem.
|Utility||Amount of Data to Transfer||Where to run utility||Notes|
|cp||< 500 GB||command line / batch script||Useful when transferring small amounts of data, available from Titan and the DTNs|
|bbcp||< 500 GB||DTN command line or batch script||Multi-streaming ability can make bbcp a faster option than cp, should be executed from DTNs only|
|fcp||> 500 GB||batch script||Faster than cp and bbcp for directory trees, can preserve lustre striping, only available from the batch scheduled DTNs|
For large transfers, transfers to the High Performance Storage System (discussed below), and when running fcp or bbcp, the DTNs should be used to prevent overloading the compute system’s login nodes.
Transfers to the High Performance Storage System
Since all areas of the shared luster filesystem are periodically swept, moving data from lustre to archival storage on High performance storage system is a necessary part of most users’ work flow. The commands
htar provide users with easy-to-use interfaces to their User Archive and Project Archive spaces on the OLCF’s HPSS-based archival storage system.
The Transferring Data with HSI and HTAR page contains examples of hsi and htar and sample workflows.