Command-Line Data Transfer Guide


Common command-line utilities, such as scp and rsync, may also be used to transfer large amounts of data to DesignSafe. Command line tools require the shortest setup time (assuming you have a compatible terminal), however are generally found challenging for first-time users. Therefore, command line transfers are only recommended in specific circumstances where other tools (see Data Transfer Guide) have been tried and found to be insufficient.


Step 1: Submit a ticket to activate Corral access for your DesignSafe/TACC Account

If you have already requested access to Corral for using Globus or another data transfer method (refer to Data Transfer Guide), you may skip this step.

To request access, go to the DesignSafe website and submit a ticket to request Corral access (https://designsafe-ci.org/help). In the body of the message say you wish to use a command-line based data transfer. Please list the Data Depot locations(s), such as My Data or a project in My Projects, where you intend to transfer files to/from. You will get a response via email when the access request is complete.


Step 2: Set up MFA using the TACC Token App

Go to TACC user portal (https://portal.tacc.utexas.edu) and log in with your DesignSafe/TACC credentials and pair a device with your account.

Full instructions are provided here: https://portal.tacc.utexas.edu/tutorials/multifactor-authentication.

Select the TACC Token App option.


Step 3: Select Transfer Utility and Perform Transfer

There are several different command-line based file transfer utilities. We detail two of them here: scp and rsync.


scp

A data transfer can be performed using the secure copy (scp) utility between any Linux, Mac, or Windows (with Window's Subsystem for Linux) machine and DesignSafe.

 

A file can be copied from your local system to the remote server by using the command:

scp <filename> <username>@data.tacc.utexas.edu:</path/to/project/directory>

where <filename> is the name of the file you wish to copy, <username> is your DesignSafe/TACC username, and </path/to/directory> is the path on Corral where you wish to send the copy of your file.To select the appropriate path on Corral see the guide for Setting the Path to DesignSafe on Corral.

 

An entire folder can be copied from your local system to the remote server by using the command:

scp -r </path/to/folder/> <username>@data.tacc.utexas.edu:</path/to/project/directory/>

where the -r indicates the copy should be recursive, </path/to/folder/> is the name of the folder you wish to copy (be sure to include the final "/", and all other terms defined previously.

 

For help execute:

scp -h

For more information execute:

man scp

 


rsync

A data transfer can also be performed using the rsync utility between any Linux, Mac, or Windows (with Window's Subsystem for Linux) machine and DesignSafe. The rsync utility is different from the scp utility as it first compares the source and destination files prior to performing the transfer and only performs a data transfer on the file(s) if they are different.

 

A file can be synced from your local system to the remote server by using the command:

rsync <filename> <username>@data.tacc.utexas.edu:</path/to/project/directory/>

where <filename> is the name of the file you wish to copy, <username> is your DesignSafe/TACC username, and </path/to/directory/> is the path on Corral where you wish to send the copy of your file. To select the appropriate path on Corral see the guide for Setting the Path to DesignSafe on Corral.

 

An entire directory can be synced from your local system to the remote server by using the command:

rsync -avtr </path/to/folder/> <username>@data.tacc.utexas.edu:</path/to/project/directory>

where -avtr will transfer the files recursively -r,  with the modification times -t, in the archive mode -a, and verbosely -v and all other terms defined previously.

 

For help execute:

rsync -h

For more information execute:

man rsync


Important: To avoid problems when transferring files to DesignSafe, name your files using standard ASCII characters, for example [a-z][A-Z][0-9][-,_], and avoid the use of special characters, for example [%,*,#,@,°].

If you have any issues transferring files using a command-line utility, please create a ticket (https://designsafe-ci.org/help).

Last Update: October 2021