DesignSafe supports multiple ways of moving data in and out of the Data Depot – which one is best depends on how you will use DesignSafe. While the web interface in the DesignSafe portal is easy for moving small numbers of modest size files, if you need to move large volumes of data over 100 MB, large numbers of files (> 25) or move directories, the Globus tools are the recommended way of moving data in and out of DesignSafe.
Globus supplies high speed, reliable, asynchronous transfers to DesignSafe. Globus is fast, for large volumes of data, as it uses multiple network sockets simultaneously to transfer data. It is reliable for large numbers of directories and files, as it can automatically fail and restart itself, and will only notify you when the transfers are completed successfully.
Of course, to get these benefits, there are a few setup steps you have to do beyond the normal Data Depot transfer process. Most of the steps you are only required to do once when you set up Globus to use for the first time. Several steps will need to be repeated each time you set up a new computer to use Globus for DesignSafe. Once you are set up, you can use Globus not only for transfers to and from DesignSafe, but also to access other cyberinfrastructure resources at TACC and around the world.
To start using Globus, you need to do three things: Enable Globus for your DesignSafe account, generate a unique identifier for all Globus services, and enroll the machine you are transferring data to/from with Globus (this can be your personal laptop or desktop, or a server to which you have access). Follow this one-time process to set up the Globus file transfer capability.
PLEASE NOTE: You must use your institution’s credentials and not your personal Google account when setting up Globus. If you use a personal account, you will encounter an issue with the endpoint (Corral).
Step 1: Submit a ticket to activate Globus access for your DesignSafe/TACC Account
Use the DesignSafe Help feature to request Globus access for your account. Simply submit a ticket here: https://designsafe-ci.org/help and in the body of the message say you wish to use Globus. Please list the Data Depot locations(s), such as My Data or a project in My Projects, where you intend to transfer files to/from. You will get a response via email when this is complete.
Step 2: Retrieve and Associate a DN to your DesignSafe/TACC Account
In order for Globus to know who you are when you move data in and out of DesignSafe from your computer, or between any other pair of systems, Globus needs a unique identifier for you, which is called a “Distinguished Name”, or DN. You can generate a DN instantly for free. To create a DN, you need to log in from some authoritative source that can verify your identity, typically your university or employer. If you already have a DN from another source, you can use that. If you do not, you can associate one with your account from many of the major universities in the world via the “CI Logon” service.
To retrieve your DN, go to https://cilogon.org in your browser. Select an Identity Provider from the drop-down list, and click "Log On" which will take you to the login screen for the Identity Provider you selected. If your university or employer is not in the list, we recommend registering for an XSEDE account as XSEDE is a CILogon Identity Provider.
After successfully authenticating at your chosen Identity Provider, you are redirected back to CILogon, where you can find your Certificate Subject that you will need to copy and paste in the next step:
/DC=org/DC=cilogon/C=US/O=University of Texas at Austin/CN=Sample Person A00000
Login to the TACC User Portal https://portal.tacc.utexas.edu with your DesignSafe/TACC account and select "Account Profile" from the main menu under the "Home" dropdown.
On the left of the page is a list of account actions, select "Manage DNs". You will be presented with a list of the DNs currently associated with your TACC account and a text field to associate a new DN to your account. Enter the Certificate Subject obtained from CILogon.org in the text field. Click the button to "Associate DN". This will associate the new DN with your account. Please note, it may take up to 30 minutes for this change to propagate to all TACC systems.
Step 3: Activate Your Desktop/Laptop as a Globus Endpoint and Transfer Files
Now that you have associated the DN with your TACC account and given the DN time to propagate to the systems (up to thirty minutes), you can activate the Globus transfer endpoints and begin transferring files.
Go to https://globus.org and log in.
Upon successful login you, you will be directed to the "File Manager" landing page.
Click on Endpoints.
Click “+ Create new endpoint” and follow the instructions to set up your desktop/laptop as an endpoint.
Enter a Display Name to identify your local endpoint like My Laptop, My Desktop at Home etc. and then click Generate Setup Key and click copy to copy the Personal Setup Key.
Download and Install the Globus Connect Personal client.
After install, open the Globus Connect Personal application. A pop menu pops up asking your setup key. Copy the setup key from the previous step to complete the setup.
Click on “File Manager”, and next click on the Collection field. You can choose "Your collections" and click on "My Laptop" to select the created endpoint to your computer.
You can now access the files on your desktop/laptop via Globus.
You can also click on Panels to look at two endpoints at the same time. In the other transfer endpoint, search for "TACC" and select TACC Corral 3.
After successfully authenticating, you will be redirected back to Globus and you will now be able to access your data on Corral:
The project-uid can be found in the URL link in the DesignSafe portal while viewing the project (see figure below).
You can transfer files between the selected endpoints. Please avoid creating directories through Globus as it will cause permission issues. Instead, you can use DesignSafe Data Depot to handle management of directories and folders.
Once the transfer is initiated, you can see the task id for the transfer being initiated.
Click activity to check status on all the transfers you have initiated.
You will also receive an email to the registered email address once the transfer is completed.