NHERI

Data Transfer Guide


DesignSafe supports multiple ways of moving data in and out of the Data Depot – which one is best depends on how you will use DesignSafe. While the web interface in the Data Depot is easy for moving small numbers of modest size files, if you need to move large volumes of data, large numbers of files (> 25) or move folders, the Globus, Cyberduck or Command Line tools are the recommended way of moving data in and out of DesignSafe.

This document details the following methods you can use to transfer your data:

  • Globus bulk data transfer
  • Cyberduck
  • Command-line Utilities
  • Data Depot browser interface
  • Box, Dropbox, and Google Drive

Large Data Transfer (> 100MB, > 25 files, > 2 folders)


Globus

See the DesignSafe Globus Data Transfer Guide

If you need automated transfers using Globus, see the Globus CLI Automated Transfer Guide.

Globus supplies high speed, reliable, asynchronous transfers to DesignSafe. Globus is fast, for large volumes of data, as it uses multiple network sockets simultaneously to transfer data. It is reliable for large numbers of directories and files, as it can automatically fail and restart itself, and will only notify you when the transfers are completed successfully. Globus tools are the recommended way of moving data in and out of DesignSafe.

Cyberduck

Cyberduck is an open source graphical user interface tool to accomplish large data transfers.

See the Cyberduck Data Transfer Guide for more information on how to use this tool.

Command-Line Transfer Utilities

Users can take advantage of some common command line utilities such as scp, sftp, rsync to achieve higher performance and to transfer large amount of data seamlessly. Web browsers have some restrictions on transferring data on large volume. Command line utilities may come in handy in those situations. TACC requires Multi-Factor Authentication (MFA) for command line access. 

To use command line transfers, you must first have an allocation on Corral (or the relevant storage resource). DesignSafe access alone is not sufficient.

scp

Data transfer from any Linux system can be accomplished using the scp utility to copy data to and from the DesignSafe. A file can be copied from your local system to the remote server by using the command:

scp filename username@data.tacc.utexas.edu:/path/to/project/directory 

localhost$ scp test.txt siva@data.tacc.utexas.edu:/corral-repl/projects/NHERI/shared/username

The above command will transfer your files/folders to your home directory.

Consult the man pages for more information on scp.

localhost$ man scp

rsync

The rsync command is a reliable method to transfer files, especially if you transfer files in stages, as it compares files automatically to make sure that files in the source and destination are the same. It is recommended for users who need checksum method since it involves calculating checksum for each file to make sure that the transfer is complete and accurate.

Below is the usage of rsync command for transferring a file named "myfile.c" from the current location on your desktop to your project directory in the DesignSafe

localhost$ rsync myfile.c username@data.tacc.utexas.edu:/corral-repl/projects/NHERI/projects/project_id

The above command will transfer your files/folders to your specified project directory.

An entire directory can be transferred from source to destination by using rsync as well. For directory transfers the options "-avtr" will transfer the files recursively ("-r" option) along with the modification times ("-t" option) and in the archive mode ("-a" option) to preserve symbolic links, devices, attributes, permissions, ownerships, etc. The "-v" option (verbose) increases the amount of information displayed during any transfer. The following example demonstrates the usage of the "-avtr" options for transferring a directory named "Nheri" from the current location on your desktop to DesignSafe's project area.

localhost$ rsync -avtr Nheri \
username@data.tacc.utexas.edu:/corral-repl/projects/NHERI/projects/project_id

For more rsync options and command details, run the command "rsync -h" or:

localhost$ man scp
localhost$ man rsync

** If rsync is used as a mode of data transfer, existing data written to the staging area will be overwritten only if the contents change. Use --ignore-existing to change this behavior. If any other data transfer protocol is used, files or folders with an existing name will be overwritten.

 

Normal Data Transfer (< 100 MB, < 25 files, < 2 folders)


Browser-based File Upload to Data Depot

User can easily transfer files and directories to their "My Data" or to specific projects via web browser.

  • Go to the Data Depot in the Workspace tab
  • Navigate to the directory where you want your data to be uploaded

  • Select the + Add button, then select File Upload

  • Select the Choose Files button and select the file(s) or folder(s) from your computer
  • Uploading sub-folders is allowed
  • Select Begin upload to upload  Once the upload is done, the window will close and you will see your data in the Data Depot

Browser-Based File Transfer within Data Depot

  • Select the file(s) or folder(s) you want to transfer

  • Select either the Move or Copy buttons in the toolbar
  • To prevent accidental loss of data, the Move button does not transfer files between root directories such as My Data and My Projects. Use the Copy button instead.

  • Once you have chosen the location: you will have 4 options
    • Select folder name: Selecting a folder will bring you into that folder, allowing you to choose a specific sub-directory to transfer the file or folder into
    • Copy/Move: This will transfer your file(s) to that folder
    • Copy Here/Move Here: This will move the file or folder to the directory you are currently viewing
    • Dropdown: Switch the root directory (Only for copying files)

Box, Dropbox, and Google Drive

DesignSafe provides user the capability to connect to your preferred cloud storage provider.

  • To setup Box, Dropbox, or Google Drive select their lefthand section in the Data Depot and follow the link to your section of Manage Account where you manage 3rd party licenses.
  • Data in Box.com or Dropbox can be copied to My Data or My Projects. You can also copy files back to Box or Dropbox from My Data or My Projects. We do not actively sync your data.
  • Original data still resides in Box or Dropbox. If you make any changes to those files in My Data or My Projects, it will not be replicated at Box or Dropbox.

 

Last update: August, 2021