Simulation

Overview

Requesting New Applications

DesignSafe regularly adds new software applications in support of natural hazards engineering research. You may contact DesignSafe by submitting a help ticket if you would like to request the addition of a software application to the Workspace.

Getting Your Own HPC Application

For those researchers with larger computational needs on the order of tens of thousands, or even millions of core-hours, or if you have a software application that we don't support in the web portal, you may request your own allocation of computing time on TACC's HPC systems. Your files can still be stored in the Data Depot, allowing you to share your research results with your team members, as well as curate and publish your findings.

Commercial/Licensed Applications

The DesignSafe infrastructure includes support for commercial/licensed software. Wile in some cases licenses can be provided by the DesignSafe project itself, not all vendors will make licenses available for larger open communities at reasonable cost. You may contact DesignSafe by submitting a help ticket if you have questions regarding a commercial software application.

ADCIRC User Guide

ADCIRC (ADvanced CIRCulation model) is a system of computer programs often used in Coastal Engineering Storm Surge research for solving time dependent, free surface circulation and transport problems in two and three dimensions. These programs utilize the finite element method in space allowing the use of highly flexible, unstructured grids. SWAN (Simulating WAves Nearshore) adds shallow water modeling. Typical research topics include:

  • prediction of storm surge and flooding
  • modeling tides and wind driven circulation
  • larval transport studies
  • near shore marine operations
  • dredging feasibility and material disposal studies

Multiple configurations and versions of ADCIRC are available in the Workspace:

  • ADCIRC
    • ADCIRC is a highly developed computer program for solving the equations of motion for a moving fluid on a rotating earth. These equations have been formulated using the traditional hydrostatic pressure and Boussinesq approximations and have been discretized in space using the finite element (FE) method and in time using the finite difference (FD) method.
    • This version of the application is ideal for smaller simulations, and runs on a single node on Frontera. Runtimes are subject to current wait times in the Frontera job queue.
  • PADCIRC
    • This is the Parallel version of the ADCIRC application and uses multiple nodes on TACC's Frontera or Lonestar6 HPC resource and is ideal for larger simulations. Runtimes are subject to current wait times in the HPC job queues.
  • PADCIRC SWAN
    • The use of the unstructured mesh version of SWAN resolves several issues previously associated with the coupling of wave and circulation models. The tightly coupled SWAN + ADCIRC paradigm allows both wave and circulation interactions to be solved on the same unstructured mesh resulting in a more accurate and efficient solution technique.
    • This version of the application uses multiple nodes on TACC's Frontera or Lonestar6 HPC resource and is ideal for larger simulations. Runtimes are subject to current wait times in the HPC job queues.

More detailed information and ADCIRC user documentation can be found at the ADCIRC website.

How to submit an ADCIRC Job in the Workspace

  • Select the appropriate ADCIRC application from the Simulation tab in the Workspace.
  • Locate your Input Directory (Folder) with your input files that are in the Data Depot and follow the onscreen directions to enter this directory in the form.
  • For the Parallel versions, enter your Mesh File into the form (usually fort.14 file).
  • Enter a maximum job runtime in the form. See guidance on form for selecting a runtime.
  • Enter a job name.
  • Enter an output archive location or use the default provided.
  • For the Parallel versions, select the number of nodes to be used for your job. Larger data files run more efficiently on higher node counts.
  • Click Run to submit your job.
  • Check the job status by clicking on the arrow in the upper right of the job submission form.

ClawPack User Guide

Clawpack (“Conservation Laws Package”) is a collection of finite volume methods for linear and nonlinear hyperbolic systems of conservation laws. Clawpack employs high-resolution Godunov-type methods with limiters in a general framework applicable to many kinds of waves.

More detailed information and CLAWPACK user documentation can be found at the Clawpack website.

How to Start a Clawpack Interactive Session in the Workspace

The Clawpack 5.4.0 suite has been installed into the DesignSafe Jupyter Hub environment. It is available for use with Python2 both from the command-line and in Jupyter notebooks. If using python2 from the terminal, you'll want to use the binaries located in '/opt/conda/envs/python2/bin'. To start using Clawpack:

  • Select the Clawpack application from the Simulation tab in the Workspace.
  • Click on Launch Jupyter.
  • Open a GeoClaw notebook (see below to access an example notebook to get you started).

Example Clawpack Use Case

An example GeoClaw notebook in Jupyter can be seen by navigating to 'community / Jupyter Notebooks / Workspace Application Sample Notebooks / GeoClaw' and opening 'GeoClaw_topotools_example.ipynb', and can be previewed and copied to your own space from here.

Dakota User Guide

The Dakota project delivers both state-of-the-art research and robust, usable software for optimization and uncertainty quantification. Broadly, the Dakota software's advanced parametric analyses enable design exploration, model calibration, risk analysis, and quantification of margins and uncertainty with computational models. The Dakota toolkit provides a flexible, extensible interface between such simulation codes (e.g. OpenSees) and its iterative systems analysis methods, which include:

  • optimization with gradient and nongradient-based methods;
  • uncertainty quantification with sampling, reliability, stochastic expansion, and epistemic methods;
  • parameter estimation using nonlinear least squares (deterministic) or Bayesian inference (stochastic);
  • and sensitivity/variance analysis with design of experiments and parameter study methods.

These capabilities may be used on their own or as components within advanced strategies such as hybrid optimization, surrogate-based optimization, mixed integer nonlinear programming, or optimization under uncertainty.

More detailed information and Dakota user documentation can be found at the Dakota website.

How to Submit a Dakota Job in the Workspace

  • Select the Dakota appication from the Simulation tab in the Workspace.
  • Locate your Input Directory (Folder) with your input files that are in the Data Depot and enter this directory in the form.
  • Enter the name of your Dakota Drive File located in your Input Directory into the form.
  • Enter a comma separated list of modules to load.
  • Enter a name for your Dakota Output File.
  • Enter a name for your Dakota Input File.
  • Enter a name for your Dakota Error File.
  • Enter a maximum job runtime in the form. See guidance on form for selecting a runtime.
  • Enter a Job name.
  • Enter an output archive location or use the default provided.
  • Select the number of nodes to be used for your job. Larger data files run more efficiently on higher node counts.
  • Click Run to submit your job.
  • Check the job status by clicking on the arrow in the upper right of the job submission form.

LS-DYNA User Guide

Ls-Dyna is a general-purpose multi-physics simulation software package. It was originated from DYNA3D, developed by John Hallquist at the Lawrence Livermore National Laboratory in 1976. The software was commercialized as LS-Dyna in 1988 by Livermore Software Technology Corporation (LSTC).

The main Ls-Dyna capabilities are:

  • 2D and 3D;
  • Nonlinear dynamics (implicit and explicit solvers);
  • Static and quasi-static analysis;
  • Thermal analysis and electromagnetic analysis;
  • Computational fluid dynamics:
  • Incompressible and compressible solver;
  • Fluid-structure interactions;
  • Smoothed-particle hydrodynamics;
  • Geotechnical and soil-structure interactions;
  • Contact algorithms;
  • Large library of elements (shell, discrete, solid, etc).

Ls-Dyna on DesignSafe

In the Workspace are available the following apps:

  • LS-PrePost: Pre-processing and post-processing of models (Workspace > Simulation >LS-DYNA> LS-Pre/Post);
  • LS-Dyna: Actual solver (version 9.1.0) – serial and parallel versions (after activation - Workspace > My Apps > LS-DYNA).

Activate Ls-Dyna on DesignSafe

DesignSafe (through TACC) has negotiated with LSTC to allow LS-DYNA access on TACC systems for academic research. Users can submit a ticket (https://www.designsafe-ci.org/help/new-ticket/) requesting LS-DYNA access and are granted access upon verification with LSTC that they have an existing academic departmental license or that you acquire such license.

A Request Activation button is also available in Workspace > Simulation > LS-DYNA:

Once activated, Ls-Dyna will appear in Workspace > My Apps tab.

How to launch LS-Dyna

Examples in this guide:

  • Launching LS-Pre/Post to generate or visualize model via DesignSafe web portal;
  • Launching a single job via DesignSafe web portal;
  • Launching batch of jobs via DesignSafe web portal;
  • Launching batch of jobs via Command Line Interface.

Launching LS-Pre/Post

  • Select the LS-Pre/Post app from the drop-drown menu at (Workspace > Simulation > LS-DYNA):

  • Fill the form with the following information:
    • Working directory: that contains the files that you want to work on;
    • Desktop resolution: select the desktop screen size for the visualization;
    • Maximum Job runtime: The maximum time you expect this job to run for. Note that after this amount of time your job will be killed by the job scheduler;
    • Job name;
    • Job output archive location (optional): location where the job output should be archived;
    • Node Count: Number of requested process nodes for the job;
    • Processors per Node: numbers of cores per node for the job. The total number of cores used is equal to NodeCount x ProcessorsPerNode.
    • Click on Run;

  • Once ready a new window will pop-up.
    • Click on Connect! and start using LS-Dyna Pre/Post.

Launching a single job via DesignSafe web portal

  • Select LS-DYNA (Serial) from the LS-DYNA app in (Workspace > My Apps > LS-DYNA):

  • Fill the form with the following information:
    • Working directory: that contains the files that you want to work on;
    • LS-DYNA Input: provide the input file name;
    • Maximum Job runtime: The maximum time you expect this job to run for. Note that after this amount of time your job will be killed by the job scheduler;
    • Job name;
    • Job output archive location (optional): location where the job output should be archived;
    • Node Count: Number of requested process nodes for the job;
    • Processors per Node: numbers of cores per node for the job. The total number of cores used is equal to NodeCount x ProcessorsPerNode;
    • Click on Run.

Launching batch of jobs via DesignSafe web portal

  • Select LS-DYNA (Parallel) from the LS-DYNA app in (Workspace > My Apps > LS-DYNA):

  • Fill the form with the following information:
    • Working directory: that contains the files that you want to work on;
    • Number of Processors: insert the number of processors required to solve each job;
    • LS-DYNA Input: provide the input file name. This file should be a text file that lists the names of all jobs to run. All jobs need to be in the same directory (see example file below);
    • Maximum Job runtime: The maximum time you expect this job to run for. Note that after this amount of time your job will be killed by the job scheduler;
    • Job name;
    • Job output archive location (optional): location where the job output should be archived;
    • Node Count: Number of requested process nodes for the batch of jobs;
    • Processors per Node: numbers of cores per node for the batch of jobs. The total number of cores used is equal to NodeCount x ProcessorsPerNode;
    • Click on Run;

Example Ls-Dyna input file for parallel jobs.

Launching batch of jobs via Command Line Interface (CLI)

  • Connect to Frontera using SSH Client. See TACC's [Data Transfer & Management Guide](https://docs.tacc.utexas.edu/hpc/frontera/):
    • Host name: frontera.tacc.utexas.edu;
    • Username and Password should be the same ones as for DesignSafe.

  • Transfer LS-Dyna k files to /scratch or /work directory on Frontera (via Globus);
  • Generate 3 files:
    • A batch file (launcherLs_Dyna.slurm) that contains all the information about the resources that you need for the parallel job and calls input files (see example file below);
    • 2 python routines to launch pylauncher that is a python-based parametric job launcher:
      • myparallellines.py: it generates a file with the list of the n jobs that you need to run (see example file below);
        • This routine will generate a my_parallel_lines file (for n=4) (see example file below);
    • launcher_file.py: it calls the application that allows to submit all the jobs in parallel. The number of cores assigned to each job is specified in cores (34 in this example). The script will call the file generated by myparallellines.py script (see example file below);

Example batch file for Ls-Dyna via CLI.

Example myprallellines.py routine.

Example my_parallel_lines file.

Example launcher_file.py routine.

  • Launch jobs using SSH client:
    • cd directory_where_your_inputs_are
    • sbatch launcherLs_Dyna.slurm (slurm file)
    • squeue –u user_name (see status of job)
  • Emails at begin and end of job will be sent to the email address provided in the batch file.
  • Once the analysis is completed, output files can be transferred using Globus (or SCP client) and used LS-PrePost (or else) to visualize them.

OpenFOAM User Guide

The OpenFOAM (Open Field Operation and Manipulation) CFD (Computational Fluid Dynamics) Toolbox is a free, open source CFD software package which has a large user base across most areas of engineering and science, from both commercial and academic organizations. OpenFOAM has an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics. It includes tools for meshing, notably snappyHexMesh, a parallelised mesher for complex CAD geometries, and for pre- and post-processing. Almost everything (including meshing, and pre- and post-processing) runs in parallel as standard, enabling users to take full advantage of high performance computing resources.

More detailed information and OpenFOAM user documentation can be found at the OpenFOAM website.

How to Submit an OpenFOAM Job in the Workspace

  • Select the OpenFOAM application from the Simulation tab in the Workspace.

  • Select the version of OpenFOAM you wanted to work with (Designsafe supports version 6 and 7).

  • Locate your Case Directory (Folder) with your input files that are in the Data Depot and follow the onscreen directions to enter this directory in the form. The following figure shows the example case in the community data.

  • Select your Solver from the dropdown menu. The workspace has options for 5 OpenFOAM solvers i.e. interFOAM, simpleFOAM, icoFOAM, potentialFOAM and olaFlow. If you need any other specfic solver please submit a ticket.

  • Choose decompostion and mesh generation options from the dropdown menu.

  • Enter maximum run time.
  • Enter a job name.
  • Enter an output archive location or use the default provided.

  • Select the number of nodes to be used for your job. Larger data files run more efficiently on higher node counts. Follow instructions given in the description for chosing number of processors.
  • Click Run to submit your job.
  • Check the job status by clicking on the arrow in the upper right of the job submission form.


OpenSees User Guide

The Open System for Earthquake Engineering Simulation (OpenSees) is a software framework for simulating the static and seismic response of structural and geotechnical systems. It has advanced capabilities for modeling and analyzing the nonlinear response of systems using a wide range of material models, elements, and solution algorithms.

One sequential (OpenSees-EXPRESS) and two parallel interpreters (OpenSeesSP and OpenSeesMP) are available on DesignSafe. Please explore the desired interpreter for more details.

OpenSees-EXPRESS

OpenSees-Express provides users with a sequential OpenSees interpreter. It is ideal to run small sequential scripts on DesignSafe resources freeing up your own machine.

OpenSeesSP

OpenSeesSP is an OpenSees interpreter intended for high performance computers for performing finite element simulations of very large models on parallel machines. OpenSeesSP is easy to use even with limited knowledge about parallel computing. It only requires minimal changes to input scripts to make them consistent with the parallel process logic. .

OpenSeesMP

OpenSeesMP is an OpenSees interpreter intended for high performance computers for performing finite element simulations with parameteric studies and very large models on parallel machines. OpenSeesMP requires understanding of parallel processing and the capabilities to write parallel scripts.

How to Submit an OpenSees Job in the Workspace

  1. Select the OpenSees application from the simulation tab in the workspace.

  2. Choose the application that is best suited for your work.

  3. Locate your OpenSees input files and TCL script in the Data Depot and follow the onscreen directions to provide your Input Directory and TCL Script in the form.

  4. To test out a tutorial case you can copy paste the link in the description for working directory as well as the TCL script name. As shown in the figure below.

  5. Enter a maximum job runtime in the form. See guidance on form for selecting a runtime.

  6. Enter a job name (Optional).

  7. Enter an output archive location or use the default provided.

  8. Node Count: Number of requested process nodes for the job.

  9. Processors per Node: numbers of cores per node for the job. The total number of cores used is equal to NodeCount x ProcessorsPerNode.

  10. Click Run to submit your job.

  11. Check the job status by clicking on the arrow in the upper right of the job submission form.

DesignSafe Tutorial: OpenSees & DesignSafe, October 31, 2018

For detailed explanation of slides below, watch the tutorial above.

Additional Resources

Examples in Community Data

  • OpenSees-EXPRESS:
  • OpenSeesSP:
    • input directory
    • input TCL file: RigidFrame3D.tcl
    • resources: 1 Node, 2 Processors
  • OpenSeesMP:
    • input directory
    • input TCL file: parallel_motion.tcl
    • resources: 1 Node, 3 Processors

Powerpoint Presentations