titan

Down since 5/27/17 10:20 pm

eos

Down since 5/27/17 11:15 pm

rhea

Down since 5/27/17 10:40 pm

hpss

Up since 4/23/17 12:00 am

atlas1

Degraded since 5/27/17 10:30 pm

atlas2

Degraded since 5/27/17 10:30 pm
OLCF User Assistance Center

Can't find the information you need below? Need advice from a real person? We're here to help.

OLCF support consultants are available to respond to your emails and phone calls from 9:00 a.m. to 5:00 p.m. EST, Monday through Friday, exclusive of holidays. Emails received outside of regular support hours will be addressed the next business day.

Transferring Data

See this article in context within the following user guides: Data | Titan

OLCF users are provided with several options for transferring data among systems at the OLCF as well as between the OLCF and other sites.

Data Transfer Nodes

Dedicated data transfer nodes are provided to OLCF users and are accessible via the load-balancing hostname dtn.ccs.ornl.gov. The nodes have been tuned specifically for wide area data transfers, and also perform well on the local area. They are recommended for data transfers as they will, in most cases, improve transfer speed and help decrease load on computational systems’ login nodes. More information on these nodes can be found on the Data Transfer Nodes page.

Local Transfers

The OLCF provides a shared-storage environment, so transferring data between our machines is largely unnecessary. However, we provide tools both to move large amounts of data between scratch and archival storage and from one scratch area to another. More information can be found on the local transfers page.

Remote Transfers

The OLCF provides several tools for moving data between computing centers or between OLCF machines and local user workstations. The following tools are primarily designed for transfers over the internet, and aren’t recommended for use transferring data between OLCF machines.

The following table summarizes options for remote data transfers:

GridFTP + GridCert GridFTP + SSH SFTP/SCP BBCP
Data Security insecure (default) / secure (w/configuration) insecure (default) / secure (w/configuration) secure insecure (unsuited for sensitive projects)
Authentication Passcode Passcode Passcode Passcode
Transfer speed Fast Fast Slow Fast
Required Infrastructure GridFTP server at remote site GridFTP server at remote site Comes standard with SSH install BBCP installed at remote site

GridFTP

GridFTP is a high-performance data transfer protocol based on FTP and optimized for high-bandwidth wide-area networks. It is typically used to move large amounts of data between the OLCF and other majors centers.
Globus is a kind of GridFTP that provides a web user-interface for initiating, managing, and monitoring GridFTP transfers between endpoints. An endpoint is the logical address of a resource or filesystem attached to a Globus Connect GridFTP server. Many institutions host their own shared Globus Connect Servers and endpoints. However, it is possible to turn any non-commercial private resource into an endpoint using the Globus Connect Personal client. Globus can also be scripted. More information for GridFTP can be found on the GridFTP page. More information for Globus can be found on the Globus page.

SFTP and SCP

The SSH-based SFTP and SCP utilities can be used to transfer files to and from OLCF systems. Because these utilities can be slow, we recommend using them only to transfer limited numbers of small files. More information on these utilities can be found on the SFTP and SCP page.

BBCP

For larger files, the multi-streaming transfer utility BBCP is recommended. The BBCP utility is capable of breaking up your transfer into multiple simultaneously transferring streams, thereby transferring data much faster than single-streaming utilities such as SCP and SFTP. Note: BBCP is not secure, but is much faster than SFTP. More information can be found on the BBCP page.