DesignSafe supports multiple ways of moving data in and out of the Data Depot – which one is best depends on how you will use DesignSafe. While the web interface in the DesignSafe portal is easy for moving small numbers of modest size files, if you need to move large volumes of data, large numbers of files (> 50) or move folders, the Globus tools are the recommended way of moving data in and out of DesignSafe.
Globus supplies high speed, reliable, asynchronous transfers to DesignSafe. Globus is fast, for large volumes of data, as it uses multiple network sockets simultaneously to transfer data. It is reliable for large numbers of directories and files, as it can automatically fail and restart itself, and will only notify you when the transfers are completed successfully.
Of course, to get these benefits, there are a few setup steps you have to do beyond the normal Data Depot transfer process. Most of the steps you are only required to do once when you set up Globus to use for the first time. Several steps will need to be repeated each time you set up a new computer to use Globus for DesignSafe. Once you are set up, you can use Globus not only for transfers to and from DesignSafe, but also to access other cyberinfrastructure resources at TACC and around the world.
To start using Globus, you need to do three things: Enable Globus for your DesignSafe account, generate a unique identifier for all Globus services, and enroll the machine you are transferring data to/from with Globus (this can be your personal laptop or desktop, or a server to which you have access). Follow this one-time process to set up the Globus file transfer capability.
Step 1: Submit a ticket to activate Globus access for your DesignSafe/TACC account
Step 2: Retrieve a Distinguished Name (DN) and add it to your DesignSafe/TACC account
Step 3: Activate your desktop/laptop as a Globus Endpoint and transfer files
PLEASE NOTE: You must use your institution’s credentials and not your personal Google account when setting up Globus. If you use a personal account, you will encounter an issue with the endpoint (Corral).
Step 1: Submit a ticket to activate Globus access for your DesignSafe/TACC Account
Use the DesignSafe Help feature to request Globus access for your account. Simply submit a ticket here: https://designsafe-ci.org/help and in the body of the message say you wish to use Globus. You will get a response via email when this is complete.
Step 2: Retrieve and Associate a DN to your DesignSafe/TACC Account
In order for Globus to know who you are when you move data in and out of DesignSafe from your computer, or between any other pair of systems, Globus needs a unique identifier for you, which is called a “Distinguished Name”, or DN. You can generate a DN instantly for free. To create a DN, you need to log in from some authoritative source that can verify your identity, typically your university or employer. If you already have a DN from another source, you can use that. If you do not, you can associate one with your account from many of the major universities in the world via the “CI Logon” service.
To retrieve your DN, go to https://cilogon.org in your browser. Select an Identity Provider from the drop-down list, and click "Log On" which will take you to the login screen for the Identity Provider you selected. If your university or employer is not in the list, we recommend registering for an XSEDE account as XSEDE is a CILogon Identity Provider.
After successfully authenticating at your chosen Identity Provider, you are redirected back to CILogon, where you can find your Certificate Subject that you will need to copy and paste in the next step:
/DC=org/DC=cilogon/C=US/O=University of Texas at Austin/CN=Sample Person A00000
Login to the TACC User Portal https://portal.tacc.utexas.edu and select "Account Profile" from the main menu under the "Home" dropdown.
On the left of the page is a list of account actions, select "Manage DNs". You will be presented with a list of the DNs currently associated with your TACC account and a text field to associate a new DN to your account. Enter the Certificate Subject obtained from CILogon.org in the text field. Click the button to "Associate DN". This will associate the new DN with your account. Please note, it may take up to 30 minutes for this change to propagate to all TACC systems.
Step 3: Activate Your Desktop/Laptop as a Globus Endpoint and Transfer Files
Now that you have associated the DN with your TACC account and given the DN time to propagate to the systems (up to thirty minutes), you can activate the Globus transfer endpoints and begin transferring files.
Go to https://globus.org and log in.
Upon successful login you, you will be directed to the "Manage Data" landing page.
Check on Endpoints.
Click “+ Globus Connect Personal endpoint” and follow the instructions to set up your desktop/laptop as an endpoint.
Enter a Display Name to identify your local endpoint like My Laptop, My Desktop at Home etc. and then click Generate Setup Key and click copy to copy the Personal Setup Key.
Download and Install the Globus Connect Personal client.
After install, open the Globus Connect Personal application. A pop menu pops up asking your setup key. Copy the setup key from the previous step to complete the setup.
Click on “Transfer Files”, then on “Endpoint”, and in the popup window click on “Administered by Me” to select your desktop/laptop.
You can now access the files on your desktop/laptop via Globus.
In the other transfer endpoint, search for "TACC" and select TACC Corral 3 and set Path to /gpfs/corral3/repl/projects/NHERI/shared/(username)/
After successfully authenticating, you will be redirected back to Globus and you will now be able to access your data on Corral.
You can transfer files between the selected endpoints.
Once the transfer is initiated, you can see the task id for the transfer being initiated.
Click activity to check status on all the transfers you have initiated
You will also receive an email to the registered email address once the transfer is completed
Task ID: b4786588-1a2d-11e7-bbc2-22000b9a448b
Task Type: TRANSFER
Source: laptop (0308590e-1972-11e7-bba3-22000b9a448b)
Destination: TACC Corral3 (3f1a4bc6-feb7-11e6-bace-22000b9a448b)