NHERI

Globus CLI Automated Transfer Guide


This guide assumes some familiarity with Command Line Interfaces (CLI).  This method is of most use to NHERI centers that need to bulk upload their data on a schedule.


Step 1: Submit a ticket to activate Corral access for your DesignSafe/TACC Account

If you have already requested access to this for using Globus or other data transfer methods, you may skip this step.

Use the DesignSafe Help feature to request Corral access for your account. Simply submit a ticket here:  https://designsafe-ci.org/help and in the body of the message say you wish to use Globus. Please list the Data Depot locations(s), such as My Data or a project in My Projects, where you intend to transfer files to/from. You will get a response via email when this is complete.


Step 2: Retrieve and Associate a DN to your DesignSafe/TACC Account

If you have not already completed this step for Globus transfers, follow the instructions found on our Globus guide:

https://www.designsafe-ci.org/rw/user-guides/globus-data-transfer-guide/#!#step2


Step 3: Activate your desktop/laptop/machine as a Globus Endpoint

If your files are stored on your institution's internal storage machine, check to see if your organization has an available Globus Endpoint already available for use for transfers.  If not, talk to your administrator about making this access available.

If you do not already have an available Globus Endpoint, follow the steps on our Globus Guide for installing Globus Personal Connect found here: https://www.designsafe-ci.org/rw/user-guides/globus-data-transfer-guide/#!#step3


Step 4: Install Globus CLI

Go here for instructions for installing the Globus CLI: https://docs.globus.org/cli

The recommended method requires a system with Python3 and the ability to run pip commands.


Step 5: Get settings for CLI Transfer

Now that all the programs are installed, we will get all the information we need to set up the automation script.

  1. First we need to get the Globus Endpoint IDs.
    • In the Globus web client, go to the Endpoints page.
    • If TACC Corral3 with CILogon Authentication is not in your list of endpoints, use the Search box at the top to search for it.
    • Click on the TACC Corral3 with CILogon Authentication endpoint.
    • On this page you will see a table of information about your endpoint.  Find the item that says "Endpoint UUID". Save this UUID for later.
    • Do the same for your endpoint.
  2. Next we need the paths for each endpoint to the data.
    • Get the path where your data will be staged for transfer.  This will be specific to your machine.
    • The second path you need is the path for the TACC Corral3 endpoint.
      • For the path, it will be one of the following:
        • To access "My Data", set Path to /corral-repl/projects/NHERI/shared/(username)/
        • To access a project in “Published”, set Path to: /corral-repl/projects/NHERI/published/PRJ-XXXX/
        • To access a project in "NEES Published", set Path to: /corral-repl/projects/NHERI/public/projects/NEES-XXXX-XXXX.groups
        • To access a project in "My Projects", set Path to /corral-repl/projects/NHERI/projects/(Uid)/

Step 6: Test Globus CLI Transfer

With the endpoint IDs and the Paths, we can now do a basic transfer with the Globus CLI.

The basic structure for a Transfer command is:

globus transfer [OPTIONS] SOURCE_ENDPOINT_ID[:SOURCE_PATH] DEST_ENDPOINT_ID[:DEST_PATH]

An example Transfer command:

globus transfer --recursive [endpoint uuid for your machine]:[path on your machine] [endpoint uuid for tacc corral3]:[path on tacc corral3 to your My Data or Project]

Fill out the variables in the example command with the information collected in the previous steps and submit the transfer.  You will get a message that tells you if your transfer command worked or not.  If your transfer command worked, you will get this message:

Message: The transfer has been accepted and a task has been created and queued for executionTask ID: [taskid]

The full reference for the Globus CLI is found here: https://docs.globus.org/cli/reference

The full reference for the Transfer command is found here: https://docs.globus.org/cli/reference/transfer  You should reference this for any other Options you may want to add to your Transfer command.


Step 7: Automate Globus Transfers

After doing a basic transfer to confirm your command was correct, you can generate a script and a cron job to automate transfers.  The script will help automate transfers and you can initate a transfer whenever you want.  The cron job is only needed if you know you will need transfers out of the same directory on a regular basis.  If you only need transfers on a infrequent basis, you can skip the cron job.

Here's an example script for automating transfers.  Note that this does hard code your DesignSafe project uuid into the script, so this assumes you are always transferring to the same DesignSafe project.

An example crontab you can use to for using cron to use this script to automatically run your transfer:

0 */6 * * * /location/of/your/globustransfer.sh > /dev/null