The OLCF Data Management Working Group helps enable data movement and data workflows with OLCF users by running tests of data transfer tools and hardware and by aidding with the development of data management documentation. All OLCF users are welcome to participate, particularly those with projects with large data movement needs. Staff from other HPC centers who have users that transfer data with OLCF are also welcome.

Transfer Tests

The following publications contain studies of data transfer using OLCF systems and may be useful for users who want to know what to expect for transfer speeds between facilities connected by Wide Area Networks (WAN).

 

HPC’s Pivot to Data, Cray Users’ Group 2014 Final Proceedings,

Performance tests and comparisons for ease of use for scp, bbcp, globus_url_copy, and globus for transfer between OLCF, ALCF and NERSC.

The Practical Obstacles of Data Transfer: Why Researchers Still Love scp, NDM ’13 Proceedings of the Third International Workshop on Network-Aware Data Management,ISBN: 978-1-4503-2522-6 doi>10.1145/2534695.2534703

Best Practices for HPC centers data transfer setup,  transfer tools test and the impact of the Luster file systems on transfer performance.

Contact Information

If you would like to participate in the OLCF Data Management working group please contact:
Suzanne Parete-Koon at [email protected]