Categories: Data Management, Data Transfer
Print this article
OLCF users are provided with several options for transferring data among systems at the OLCF as well as between the OLCF and other sites.
Data Transfer Nodes
Dedicated data transfer nodes are provided to OLCF users and are accessible via the load-balancing hostname
dtn.ccs.ornl.gov. The nodes have been tuned specifically for wide area data transfers, and also perform well on the local area. They are recommended for data transfers as they will, in most cases, improve transfer speed and help decrease load on computational systems’ login nodes. More information on these nodes can be found on the Data Transfer Nodes page.
The OLCF provides a shared-storage environment, so transferring data between our machines is largely unnecessary. However, we provide tools both to move large amounts of data between scratch and archival storage and from one scratch area to another. More information can be found on the local transfers page.
The OLCF provides several tools for moving data between computing centers or between OLCF machines and local user workstations. The following tools are primarily designed for transfers over the internet, and aren’t recommended for use transferring data between OLCF machines.
The following table summarizes options for remote data transfers:
|GridFTP + GridCert||GridFTP + SSH||SFTP/SCP||BBCP|
|Data Security||insecure (default) / secure (w/configuration)||insecure (default) / secure (w/configuration)||secure||insecure (unsuited for sensitive projects)|
|Required Infrastructure||GridFTP server at remote site||GridFTP server at remote site||Comes standard with SSH install||BBCP installed at remote site|
GridFTP is a high-performance data transfer protocol based on FTP and optimized for high-bandwidth wide-area networks. It is typically used to move large amounts of data between the OLCF and other majors centers.
Globus is a kind of GridFTP that provides a web user-interface for initiating, managing, and monitoring GridFTP transfers between endpoints. An endpoint is the logical address of a resource or filesystem attached to a Globus Connect GridFTP server. Many institutions host their own shared Globus Connect Servers and endpoints. However, it is possible to turn any non-commercial private resource into an endpoint using the Globus Connect Personal client. Globus can also be scripted. More information for GridFTP can be found on the GridFTP page. More information for Globus can be found on the Globus page.
SCP utilities can be used to transfer files to and from OLCF systems. Because these utilities can be slow, we recommend using them only to transfer limited numbers of small files. More information on these utilities can be found on the SFTP and SCP page.
For larger files, the multi-streaming transfer utility
BBCP is recommended. The
BBCP utility is capable of breaking up your transfer into multiple simultaneously transferring streams, thereby transferring data much faster than single-streaming utilities such as
BBCP is not secure, but is much faster than
SFTP. More information can be found on the BBCP page.