titan

Up since 11/8/17 02:45 pm

eos

Up since 11/14/17 11:20 pm

rhea

Up since 10/17/17 05:40 pm

hpss

Up since 11/20/17 09:15 am

atlas1

Up since 11/15/17 07:25 am

atlas2

Up since 11/27/17 10:45 am
OLCF User Assistance Center

Can't find the information you need below? Need advice from a real person? We're here to help.

OLCF support consultants are available to respond to your emails and phone calls from 9:00 a.m. to 5:00 p.m. EST, Monday through Friday, exclusive of holidays. Emails received outside of regular support hours will be addressed the next business day.

GridFTP with Globus

See this article in context within the following user guides: Data

Using Globus

Globus provides a web user-interface for initiating, managing, and monitoring GridFTP transfers between endpoints.

An endpoint is the logical address of a resource or filesystem attached to a Globus Connect GridFTP server. Many institutions host their own shared Globus Connect Servers and endpoints. However, it is possible to turn any non-commercial private resource into an endpoint using the Globus Connect Personal client.

Transferring Data with Globus Online

In the browser of your choice, visit https://www.globus.org.

  1. Log in to Globus with your Globus identity. Most new Globus users should create a traditional stand-alone Globus ID by following the “Sign up” link. Returning OLCF users should generally follow the “Globus ID” link to login.
    Only users who have an ORNL UCAMS institutional ID may choose “Oak Ridge National Lab” from the dropdown menu.

    See the Globus accounts FAQ for more information.

    Login to the Globus webapp

    Login to the Globus webapp using an existing Globus ID, linked institutional ID, or new Globus account.

  2. Once logged in to the Globus webapp, select “Manage Data” and then “Transfer Files” from blue menu bar.

    Choose Transfer Files from navigation menu

    Choose Transfer Tiles from the navigation menu.

  3. Enter an OLCF endpoint into one of the two Endpoint fields. Using the endpoint selection dialog that appears, enter OLCF Atlas (or the alternate name olcf#dtn_atlas) to choose the OLCF as an endpoint.
    Select Endpoint

    Select an OLCF endpoint in one of the two available fields.

    Workflows established prior to February 2016 may have used the now-discontinued olcf#dtn endpoint. This endpoint should no longer be used. Questions about migrating legacy workflows can be directed to help@olcf.ornl.gov.
  4. Globus must request permission from you via the OLCF to access your files. Press the Continue button. You will be redirected to our authentication page at “myproxy*.ccs.ornl.gov”. Enter your OLCF username in the “username” box and your OLCF SecurID passcode in the “Passcode” box. Upon success, you will be returned to the Globus web interface.
    OLCF OAuth page

    Activating an endpoint using only your OLCF credentials requires a browser to authenticate the OAuth request from Globus.

    The endpoint lifetime is 72 hours. If the endpoint authentication expires before the transfer is complete, the transfer will automatically resume the next time you reactivate the endpoint.
  5. Enter the path to your data (for example /tmp/work/username) in the “path: window. Soon you should see a list of the files in that directory appear in the window below.
  6. Repeat this process in the other endpoint window with the endpoint at the other end of your transfer.
  7. Select the files you want to transfer by clicking on them. Use the arrows in the middle of the page to do the transfer.Screen Shot 2013-06-06 at 1.48.05 PM
  8. Globus will give you a message at the top of the page about the status of your transfer and send you an email when your transfer is complete.

Reactivating an Expired Endpoint

If the endpoint or proxy expires before the transfer is complete, the transfer will automatically resume the next time you activate the endpoint.

To reactivate an expired endpoint, choose “Manage Endpoints” from the Globus web interface. Select the OLCF endpoint you wish to reactivate and choose the “Activate” tab. Press the “Reactivate Now” button and enter your OLCF credentials to approve the request by Globus to access your account.

Reactivate endpoint

Reactivate an endpoint under the Manage Endpoints section.

Globus Online Command Line Interface

Globus Online also provides a scriptable command-line interface available via SSH at cli.globusonline.org using your Globus account credentials. Complete information about cli.globusonline.org can be found in the official Globus documentation.

To use the CLI interface, you must first generate an SSH public-private key pair on the host from which you will use the interface. From a terminal, call

$ ssh-keygen -t rsa -b 4096 -C "Globus key on $HOST" -f $HOME/.ssh/id_rsa.globus
It is highly recommended that all your SSH keys are protected by passphrases. Passphrase-protected keys can be used in conjunction with an SSH-agent for convenience.
Compromised passphrase-less keys linked to Globus will allow read-write access to all of your activated endpoints.

Add the public key to your Globus ID’s list of authorized keys. From the web-UI, click on the Account menu link, then choose “manage SSH and X.509 keys”, then “Add a New Key”. Give the key any alias, choose the SSH Public Key Type, paste the full contents of $HOME/.ssh/id_rsa.globus.pub into the body field and click “Add Key”.

To use the interface, start an SSH session as your globus ID username with

$ ssh -i $HOME/.ssh/id_rsa.globus ${GLOBUS_UNAME}@cli.globusonline.org

This command will place you into an interactive console from which globus transfer management commands can be issued. Calling help will list all of the available commands. Full documentation for each command is available through man $COMMAND.

By encapsulating the SSH invocation into a shell function or alias and using an SSH-agent or passphrase-less key, it is possible to write convenient shell scripts for managing Globus transfers. The script below uses the Globus Online Tutorial Endpoints go#ep1 and go#ep2, which are available to all Globus users for practice, to demonstrate basic operations.

#!/bin/bash
#
# This script demos the simplest way to automate Globus Online transfers
# using the ssh://cli.globusonline.org interface.
#
#==============================================================================

# Edit these as needed for individual use
PUBKEY="$HOME/.ssh/id_rsa.globus"
GLOBUS_UNAME="FIXME"

# Function to simplify remote Globus command invocations.
gocli() {
  ssh -i ${PUBKEY} ${GLOBUS_UNAME}@cli.globusonline.org "$@"
}

# Print available commands. Man pages can be read by starting an interactive
# session over ssh using `ssh ${GLOBUS_UNAME}@cli.globusonline.org`
gocli help

# Activate the endpoints.
# endpoint-activate returns 0 if active or successfully activated.
# Some endpoints may be interactively activated, but not the OLCF's.
# It is a good practice to exit the script on activation problems when the
# script is run non-interactively.
# TODO - Add a trap or timeout if this script is run non-interactively
#        against endpoints that can be interactively activated.
gocli endpoint-activate "go#ep1"
[ "$?" = 1 ] && exit 1

gocli endpoint-activate "go#ep2"
[ "$?" = 1 ] && exit 1

# Make destination dirs - this is not strictly necessary, just showing off
# `mkdir`.
if ! $(gocli ls "go#ep2/~/simple_demo" > /dev/null 2>&1); then
  gocli mkdir "go#ep2/~/simple_demo"
fi

# List the SRC and DST folder contents.
gocli ls -l "go#ep1/share/godata"
gocli ls -l "go#ep2/~/simple_demo"

# Bulk file transfer:
# Constuct array of INPUTLINE(s) from ls output on src dir:
DATA_FILES=( $(gocli ls "go#ep1/share/godata/*.txt") )
{
for i in "${!DATA_FILES[@]}"; do
  f="${DATA_FILES[$i]}"
  echo "go#ep1/share/godata/$f go#ep2/~/simple_demo/bulk/$f"
done
# Pipe array into transfer command.
} | gocli transfer -s 3 --label="scripted_bulk_xfer_demo"

# Recursive transfer:
gocli transfer -s 3 --label="scripted_recursive_xfer_demo" -- \
  "go#ep1/share/godata/" \
  "go#ep2/~/simple_demo/recursive/" -r

# Print the status of the last two transfers:
# See `gocli man status` for format options to make parsing easier.
gocli status -a -l 2