DesignSafe supports multiple ways of moving data in and out of the Data Depot – which one is best depends on how you will use DesignSafe. While the web interface in the Data Depot is easy for moving small numbers of modest size files, if you need to move large volumes of data, large numbers of files (> 25) or move folders, the Globus, Cyberduck or Command Line tools are the recommended way of moving data in and out of DesignSafe.
This document details the following methods you can use to transfer your data:
If you need automated transfers using Globus, see the Globus CLI Automated Transfer Guide.
Globus supplies high speed, reliable, asynchronous transfers to DesignSafe. Globus is fast, for large volumes of data, as it uses multiple network sockets simultaneously to transfer data. It is reliable for large numbers of directories and files, as it can automatically fail and restart itself, and will only notify you when the transfers are completed successfully. Globus tools are the recommended way of moving data in and out of DesignSafe.
Cyberduck is an open source graphical user interface tool to accomplish large data transfers.
See the Cyberduck Data Transfer Guide for more information on how to use this tool.
Users can take advantage of some common command line utilities such as scp, sftp, rsync to achieve higher performance and to transfer large amount of data seamlessly. Web browsers have some restrictions on transferring data on large volume. Command line utilities may come in handy in those situations. TACC requires Multi-Factor Authentication (MFA) for command line access.
To use command line transfers, you must first have an allocation on Corral (or the relevant storage resource). DesignSafe access alone is not sufficient.
Data transfer from any Linux system can be accomplished using the scp utility to copy data to and from the DesignSafe. A file can be copied from your local system to the remote server by using the command:
scp filename firstname.lastname@example.org:/path/to/project/directory localhost$ scp test.txt email@example.com:/corral-repl/projects/NHERI/shared/username
The above command will transfer your files/folders to your home directory.
Consult the man pages for more information on scp.
localhost$ man scp
rsync command is a reliable method to transfer files, especially if you transfer files in stages, as it compares files automatically to make sure that files in the source and destination are the same. It is recommended for users who need checksum method since it involves calculating checksum for each file to make sure that the transfer is complete and accurate.
Below is the usage of rsync command for transferring a file named "myfile.c" from the current location on your desktop to your project directory in the DesignSafe
localhost$ rsync myfile.c firstname.lastname@example.org:/corral-repl/projects/NHERI/projects/project_id
The above command will transfer your files/folders to your specified project directory.
An entire directory can be transferred from source to destination by using rsync as well. For directory transfers the options "-avtr" will transfer the files recursively ("-r" option) along with the modification times ("-t" option) and in the archive mode ("-a" option) to preserve symbolic links, devices, attributes, permissions, ownerships, etc. The "-v" option (verbose) increases the amount of information displayed during any transfer. The following example demonstrates the usage of the "-avtr" options for transferring a directory named "Nheri" from the current location on your desktop to DesignSafe's project area.
localhost$ rsync -avtr Nheri \ email@example.com:/corral-repl/projects/NHERI/projects/project_id
rsync options and command details, run the command "rsync -h" or:
localhost$ man scp localhost$ man rsync
** If rsync is used as a mode of data transfer, existing data written to the staging area will be overwritten only if the contents change. Use
--ignore-existing to change this behavior. If any other data transfer protocol is used, files or folders with an existing name will be overwritten.
User can easily transfer files and directories to their "My Data" or to specific projects via web browser.
DesignSafe provides user the capability to connect to your preferred cloud storage provider.
Last update: August, 2021