Manual 01 for ParaDiS
How to obtain and run ParaDiS
Sylvie Aubry and Wei Cai
Contents |
What is ParaDiS and how to obtain a copy
ParaDiS is a free large scale dislocation dynamics simulation code to study the fundamental mechanisms of plasticity. It was originally developed at the Lawrence Livermore National Laboratory. It is written in C (with a little C++) and uses the MPI library for communication between processors. It runs routinely on 100-1000 processors and scalability on 132,000 processors of BlueGene/L has been demonstrated.
You can go to http://paradis.stanford.edu to obtain a free copy of the source code of ParaDiS. You will need to requestion an account by submitting a username, a password and a brief description of yourself and your research interests. Once your account request is approved, you will receive a confirmation email and you can download the ParaDiS source code. The same web site also contains the documentation of the ParaDiS code, including tutorials and reference manuals.
The source file you obtain should look something like pub-dd3d.v2.0.tar.gz. It is recommended that you create a ~/Codes directory and extract the source code there.
cd ~/Codes tar zxvf pub-dd3d.v2.0.tar.gz cd dd3dv2.0
This should create a ~/Codes/dd3dv2.0 directory. We will refer to it as $ParaDiS in the following. Check that you have the following directories within your $ParaDiS directory: Inputs, Tests, Tools, Utilities.
How to compile ParaDiS
Go to the $ParaDiS directory. You may need to edit the makefile.setup file. The most important variables are SYS and MODE. SYS specifies the type of your computer and MODE can be either SERIAL or PARALLEL. For example, suppose you want to compile ParaDiS on a Linux/Intel workstation with 1 processor. You want to set SYS = i386 and MODE = SERIAL in the makefile.setup file. Then, you can just type
make dd3d
If the compilation is successful, this creates the main executable dd3d in your $ParaDiS directory. If you type
make
instead, then in addition to dd3d, a few useful executables will be created in the $ParaDiS/Utilities directory.
If there are some problems during compilation, you should then look into the makefile.setup file, and check whether the lines CC_SERIAL.i386, CPP_SERIAL.i386, CCFLAG_SERIAL.i386, CPPFLAG_SERIAL.i386 correctly specify the C/C++ compilers on your computer.
You can compile ParaDiS on Windows if [cygwin] installed. Set SYS = cygwin in makefile.setup.
If you are using Ubuntu Linux or GCC and are unable to compile, please see the following guide for instructions on installing version 2.2.
Issues with Nvidia Linux graphics card driver
The X11 display has consistently crashed when running ParaDiS versions 2.0 and 2.2 and MD++ on a Dell Latitude D630 with an Nvidia Quadro 130M running Ubuntu 7.10. The window would render correctly for a few seconds, but then freeze. However, the simulation would still continue running. This error appears to be unique to Nvidia cards, because ParaDiS has been successfully ran on other Ubuntu machines. After hours spent reconfiguring the xorg.conf and searching for other libraries, the only solution found was to not use the closed-source Nvidia driver. Instead, the open-source nv driver had to be installed. This effectively renders the graphics card useless, and is thus not an ideal solution. If you have have a similar problem contact William Cash with the details of your system and any solution you might have found.
How to run a single-CPU ParaDiS simulation
Using the dd3d executable compiled as described above, we can run a few test cases in the Tests directory. Each ParaDiS run is described in two input files: a .cn file (specifying simulation control parameters) and a .data with the same name (specifying the dislocation structure). For example, we can run
./dd3d Tests/frank_read_src.cn
In this run, ParaDiS will automatically look for file Tests/fread_read_src.data to read the dislocation (nodal) structure. The output for this run should look something like:
*********************************************************** **** **** Time of run: Fri Nov 30 20:37:40 2007 **** Executable name: dd3d **** Executable path: . **** Working dir: .... **** Execution host: .... i686 **** User name: Wei Cai **** Number of tasks: 1 **** Command line: Tests/frank_read_src.cn **** *********************************************************** Initialize: Parsing control file Tests/frank_read_src.cn Initialize: Control file parsing complete ReadNodeDataFile: dataFile = Tests/frank_read_src.data node count: 3 ... chdir successful on all tasks. DD3dInit finished cycle=1 realdt=5.953988e-10 timeNow=1.511247e-06 Fri Nov 30 20:37:43 2007 cycle=2 realdt=7.144786e-10 timeNow=1.511962e-06 Fri Nov 30 20:37:43 2007 cycle=3 realdt=8.573743e-10 timeNow=1.512819e-06 Fri Nov 30 20:37:43 2007 ...
After DD3dInit finishes, ParaDiS prints one line every simulation step (cycle), which includes the cycle number, physical time step (realdt), physical time (timeNow), and computer wall time. Notice that the physical time advance for each time step is not a constant during the simulation.
By default, a window will open spontaneously, which displays the evolution of the dislocations. The setting for this window is specified in the win.script file. In particular, the window will be disabled if enable_window = 0. By default enable_window = 1 is set in this file. You can rotate the 3-dimensional dislocation structure by dragging your mouse in the display window. You can also zoom in and out, as well as save a snapshot into file. Press F1 in the window, and a list of short-keys will be printed in your terminal. For example, the F9 key saves the current window into a .gif file and the F10 saves a .ps file.
ParaDiS input and output
The content of ParaDiS control files, e.g. Tests/frank_read_src.cn look something like this:
### DD3d configuration file (use -*-shell-script-*- format) # # Sample control file to be used in conjunction with # the frank_read_src.data file. # dirname = "Tests/frank_read_src_results" Rijmfile = "Inputs/Rijm.cube.out" RijmPBCfile = "Inputs/RijmPBC.cube.out" numXdoms = 1 numYdoms = 1 numZdoms = 1 ...
Anything following the pound sign # is a comment. The values of the parameters are specified by the syntax var_name = value. A list of all ParaDiS input variables is available [here]. In the following we describe a few of the most important ones.
For single CPU runs, numXdoms, numYdoms and numZdoms should all be set to 1. For parallel runs, the total number of CPUs must match numXdoms * numYdoms * numZdoms. maxstep specifies the total number of time steps (cycles) for the simulation.
dirname specifies the directory in which all output files are created. Simulation output styles are specified by groups of variables. For example, saveprop, saveprop_freq, saveprop_dt, saveprop_dt, saveprop_time specifies how frequent and how detailed the simulation properties should be saved. The simulation property files are saved in the properties sub-directory, which contains the stress-strain curve (stress_Total_strain file) and the dislocatoin density (density file). Similarly, gnuplot, gnuplot_freq, gnuplot_counter, gnuplot_dt, gnuplot_timie together specify whether and how ParaDiS will output snapshots of dislocation structures in the [gnuplot] format.
Within the directory specified by dirname, the restart sub-directory contains restart files, e.g. rs0001 (control file) and rs0001.data (nodal data file), which are saved periodically during the simulation. The restart and restart.data files are saved at the end of the simulation. The timers sub-directory contains computational time information for different parts of the program.
How to run a parallel ParaDiS simulation
Compiling and running ParaDiS in parallel depends on the specific settings of your computer cluster. Here we explain the main steps using our own cluster [mc-cc.stanford.edu] as an example.
First, ParaDiS need to be compiled in the parallel mode. You need to set MODE = PARALLEL in makefile.setup. You also need to choose the correct SYS or write your own to use the correct MPI compiler. For example, we wrote our own settings for SYS = mc-cc in the makefile.sys file. You may try SYS = i386 or SYS = linux if you have an Intel cluster.
In a shared cluster, you need to submit the parallel job into a queue, usually by the qsub command, such as
qsub run_dd3d.qsub
where run_dd3d.qsub may look like the following
#!/bin/bash #PBS -N my-paradis-run #PBS -j oe #PBS -l nodes=2:ppn=4,walltime=48:00:00 #PBS -V ### --------------------------------------- ### BEGINNING OF EXECUTION ### --------------------------------------- echo The master node of this job is `hostname` echo The working directory is `echo $PBS_O_WORKDIR` echo This job runs on the following nodes: echo `cat $PBS_NODEFILE` ncpu=`cat $PBS_NODEFILE | wc -w` echo "Number of processors = $ncpu " ### end of information preamble cd $PBS_O_WORKDIR echo $PWD ./dd3d Outputs/myrun/restart.cn > dd3d_output
In this example, ParaDiS will re-run a restart file (saved from a previous simulation) in the Outputs/myrun directory, using 8 CPUs. The total number of domains in the input file has to be modified to match this number, e.g. numXdoms = 2, numYdoms = 2, numZdoms = 2.
Usually you need to specify some environment variables for parallel programs to run successfully. For example, you may need to add the following lines to your ~/.bash_profile file.
PATH=$PATH:/share/apps/lib LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/lib