Globus CLI Automated Transfer Guide


Globus provides a command line interface (CLI), for those who need to perform automated data transfers. This data transfer method will likely be of most use to NHERI centers that need to bulk upload their data on a schedule.


Step 1: Submit a ticket to activate Globus access for your DesignSafe/TACC Account

Create a new ticket on DesignSafe (https://designsafe-ci.org/help) to request Globus access for your account. 

In the body of the message indicate you wish to use Globus and list the Data Depot location(s), such as My Data or a project in My Projects, where you intend to transfer files to/from.

You will get a response via email when the activation is complete.


Step 2: Create a Distinguished Name (DN)

Detailed instructions for creating a DN can be found in Step 2 of the Globus Data Transfer Guide.

If you have already completed the DN creation process and associated it with your DesignSafe/TACC account you may skip to Step 4.


Step 3: Associate your DN with your DesignSafe/TACC Account

Detailed instructions for associating your DN with your DesignSafe/TACC account can be found in Step 3 of the Globus Data Transfer Guide.


Step 4: Activate Your Desktop/Laptop as a Globus Endpoint and Connect

If the data you wish to transfer is located on your local machine, follow Step 4 of the Globus Data Transfer Guide to create a personal endpoint.

If the data you wish to transfer is located on a server operated by your organization and does not already have a Globus Endpoint available, talk to your system administrator about creating one.


Step 5: Install the Globus CLI

Follow the instructions provided by Globus for installing the CLI (https://docs.globus.org/cli)

Note the recommended installation method requires a system with Python3 and the ability to run pip commands.


Step 6: Settings for CLI Transfer

With the Globus CLI successfully installed on our local machine, we must now determine the endpoint information for DesignSafe.

Go to the Globus web interface > select the search bar.

Select Globus Search Bar

Search for TACC Corral3 with CILogon Authentication > click the three vertical dots to the right to view endpoint details.

Search and Select Three Dots

Copy the Endpoint UUID and store for later reference.

Copy UUID

Repeat the process above to attain the UUID for your local endpoint.


Step 7: Test Globus CLI Transfer

With the endpoint IDs, we can now do a test transfer with the Globus CLI.

Start by authenticating by entering the following:

globus login

This will prompt you to authenticate through your web-browser and grant permissions to Globus-CLI.

Next, we create our transfer command. The basic structure follows:

globus transfer [OPTIONS] SOURCE_ENDPOINT_ID[:SOURCE_PATH] DEST_ENDPOINT_ID[:DEST_PATH]

An example transfer command:

globus transfer --recursive [endpoint uuid for your machine]:[path on your machine] [endpoint uuid for tacc corral3]:[path on tacc corral3 to your My Data or Project]

Fill out the variables in the example command with the UUIDs and paths and submit the transfer. If you are unsure of the appropriate Corral3 path, please refer to the guide on Setting the Path to DesignSafe on Corral for more information.

You will get a message stating whether your transfer was successful or not. If it was successful, you will receive the message:

Message: The transfer has been accepted and a task has been created and queued for executionTask ID: [taskid]

The full reference for the Globus CLI can found here: https://docs.globus.org/cli/reference.

The full reference for the transfer command, including information on additional options that may be useful to you, can be found here: https://docs.globus.org/cli/reference/transfer.


Step 8: Create an Automatic Transfer Script

We will now create a shell script to store the transfer details (i.e., UUIDs and paths) and globus-cli syntax to allow us to quickly and reliably initiate future transfers.

Below is an example script you can modify for your transfers. Note that this does hard code the UUIDs and paths and therefore assumes you are always transferring to and from the same locations.

#!/bin/bash

GLOBUS_CLI_INSTALL_DIR="$(python -c 'import site; print(site.USER_BASE)')/bin"

export PATH="$GLOBUS_CLI_INSTALL_DIR:$PATH"

# Globus endpoints
epid1 = ENTER_YOUR_GLOBUS_CONNECT_ENDPOINT_ID
epid2 = ENTER_THE_TACC_CORRAL_ENDPOINT_ID

# Data paths
local_path = PATH_TO_YOUR_DATA_ON_LOCAL_MACHINE
remote_path = PATH_TO_YOUR_DATA_ON_CORRAL3

# End points
ep1 = $epid1:$local_path
ep1 = $epid2:$remote_path

# Time stamped label for transfer
label=$(data + "%Y&m&d_%H%M%S")
label=$"YourLabelHere_${label}"

# Run transfer
globus transfer --recursive --label $label "$ep1" "$ep2"

Step 9: Automate Script Execution with cron

To automate the transfer we wil use the Linux scheduling utility cron to call our transfer script on a specified schedule.

An example cron table entry that you can use to automatically run your transfer every six hours is listed below:

0 */6 * * * /location/of/your/globustransfer.sh > /dev/null


Important: To avoid problems when transferring files to DesignSafe, name your files using standard ASCII characters, for example [a-z][A-Z][0-9][-,_], and avoid the use of special characters, for example [%,*,#,@,°].

If you have any issues transferring data to DesignSafe using the Globus CLI, please create a ticket (https://designsafe-ci.org/help).

Last Update: October 2021