Written by Jie Yin
Here is a practical guide to ParaDiS corresponding to the latest version. It is especially useful for beginners with latest command templates and programming tips.
While reading this document, don't forget to check out paradis.stanford.edu for a complete and searchable table of parameters in ParaDiS code (some parameter names are not up-to-date). Detailed descriptions can be found in the official documentation ParaDiSInfo.txt.
Contents |
How to obtain the latest ParaDiS codes
It is highly recommended to create directory Codes under your home directory. Currently ParaDiS codes are maintained through Subversion. For current group members or collaborators, you can obtain the latest version of ParaDiS by typing the following,
cd ~/Codes svn co https://micro.stanford.edu/svn/ParaDiS [./dirname]
Directory ~/Codes/ParaDiS (or ~/Codes/dirname) will be created with latest checked out revision. It is your svn local repository. You can update your local ParaDiS codes by simply typing
cd ~/Codes/ParaDiS svn update
If you create a new file called newfile.c, you can add it to your local repository ~/Codes/ParaDiS
$ svn add newfile.c
or commit the change to ParaDiS codes after some modifications
$ svn commit newfile.c
How to compile ParaDiS
Before compilation, you need to modify the settings in makefile.setup, which is under directory ~/Codes/ParaDiS. Currently many system types are supported. In the first few lines, you can see following lines
SYS = wcr MODE = SERIAL #MODE = PARALLEL #XLIB_MODE = OFF XLIB = MODE = ON
SYS defines the system type. Detailed descriptions can be found in file makefile.sys. You can also change to linux if it best described your system type.
MODE is the execution mode. PARALLEL is to enable compile-time support for MPI and multi-CPU support and SERIAL is for single-CPU compilation. Usually SERIAL is used for debugging and visulization purpose. For computation intensive jobs, you should choose PARALLEL mode.
XLIB_MODE = ON is only valid when you want to enable xwindow plotting capability (at the same time MODE = SERIAL ). Otherwise you should set it OFF.
After saving your settings, you can start compilation by simply typing the following,
$ make
If you want to compile faster on a multi-CPU machine, you can use command
$ make -j8
where 8 means the number of processors used for compilation.
If you want to re-compile ParaDiS, e.g. after some modification to source files or system settings, you need to "make clean" before "make"
$ make clean
How to run a test case
To run a DD simulation, you need to prepare .ctrl file and .data file. .ctrl file includes controlling parameters needed for the simulation while .data file usually describe the geometric parameters of the simulation system, e.g. cell size, nodal coordinates. To run a test case, the basic command is shown as following
bin/paradis -d runs/testfile.data runs/testfile.ctrl
The executable is paradis. When -d is used, your .data file does not have to be in the same directory or of the same name as your .ctrl file. In most cases, people create the two files with same names and put them under the same directory, then you can use
bin/paradis runs/testfile.ctrl
How to set up data-file
In most .data files, following structures exist
dataFileVersion = 4
numFileSegments = 1
minCoordinates = [
-4.000000e+003
-4.000000e+003
-4.000000e+003
]
maxCoordinates = [
4.000000e+003
4.000000e+003
4.000000e+003
]
nodeCount = 192
dataDecompType = 1
dataDecompGeometry = [
1
1
1
]
domainDecomposition =
-4.000000e+003
-4.000000e+003
-4.000000e+003
4.000000e+003
4.000000e+003
4.000000e+003
#
# Primary lines: node_tag, x, y, z, num_arms, constraint
# Secondary lines: arm_tag, burgx, burgy, burgz, nx, ny, nz
#
# length in unit of burgMag
nodalData =
0,0 3.52197261266669e+003 -2.10366848600532e+003 8.35974608889734e+002 2 7
0,1 -0.707107 0.707107 0 1 1 1
0,15 0.707107 -0.707107 0 -0.408248 -0.408248 0.816497
...
You need to set your own data file parameters, usually including minCoordinates, maxCoordinates, nodeCount, domainDecomposition and nodalData. The nodalData is a long array of data. You may want to create it from Matlab if there exists too many nodes in your simulation box.
How to set up ctrl-file
The structure of the ctrl file is explained by parts.
1 First is the basic setup information for the simulation,
#Directory to write output files dirname = "outputs/filename" #Input files for PBC image stresses Rijmfile = "inputs/Rijm.cube.out" RijmPBCfile = "inputs/RijmPBC.cube.out" #Total simulation steps maxstep = 100 #The total number of CPUs should be numXdoms * numYdoms * numZdoms numXdoms = 1 numYdoms = 1 numZdoms = 1 #Cells for dislocation grouping (cannot be less than 3) numXcells = 3 numYcells = 3 numZcells = 3 #Fast multipole method specs. fmEnabled = 0 #disable fast multipole fmMPOrder = 2 fmTaylorOrder = 4 fmCorrectionTbl = "Inputs/fm-ctab.m2.t4.dat"
You can set the directory name where you want to store the output. You can also set the total number of simulation steps maxstep. As for numXdoms, numYdoms and numZdoms, note that the product should be the total number of CPUs. This setting is crucial for parallel jobs. When all of them as set as 2, the total number of CUPs is 8. This should be consistant with the pbs file setting, which will be discussed in the end of this manual. If you want to enable fast multipole method (FMM), you need to set fmEnabled = 1 and assign the path for the corresponding table.
2 Next is some settings on geometry, material parameters etc
timestepIntegrator = "trapezoid" #Boundary conditions xBoundType = 0 yBoundType = 0 zBoundType = 0 #Fundamental length unit burgMag = 2.556120e-010 #Elastic constants shearModulus = 40000000000.000000 pois = 0.369000 #Mobility law function mobilityLaw = "FCC_0" enforceGlidePlanes = 1 enableCrossSlip = -1 MobScrew = 1.000000e+005 MobEdge = 1.000000e+005 MobClimb = 1.000000e-002 #Discretization maxSeg = 200.000000 minSeg = 10.000000 #Maximum nodal displacement at each time step rmax = 100 #Error tolerance in determining time step rTol = 0.721688 #Maximum time step maxDT = 1 #Core cut-off radius rc = 2.886751 #Core energy Ecore = 0.000000e+000 #Turn on elastic interaction elasticinteraction = 1
Variable xBoundType defines the type of problem space boundaries in the X dimension. Valid values are 0, 1 and 2 respectively for periodic, free surface, and reflecting.
The value of burgMag is the unit of length for all length variables in ParaDiS simulation, such as nodal positions and Burgers vectors (in data file, those data should be expressed in terms of the same burgMag)
The setting for mobility law is important, which should be consistent with the system you simulate. Cross-slip is enabled only when enableCrossSlip = 1. You need to adjust some of the geometric parameters to fit your problem.
3 Third is the loading conditions
#Applied stress in Pa (xx,yy,zz,yz,zx,xy) appliedStress = [ 0 0 0 0 0 0 ] #Constant Strain rate loadType = 1 edotdir = [ 0.0 0.0 1.0 ] eRate = 100.0
loadType defines the type of load on the system. Usually we set it as 0 (creep, i.e. constant external stress) or 1 (constant strain rate). For detailed explanations for value 2, 3 and 4, please refer to the variable table. When loadType = 0, you need to set appliedStress(zero vector for relaxation). When loadType = 1, you need to define strain rate direction edotdir and strain rate eRate.
4 And the last part is settings on saving,
#Save files savecn = 1 savecnfreq = 100 #Save Properties saveprop = 1 savepropfreq = 1 #winDefaultsFile = "inputs/paradis.xdefaults"
By setting savencn = 1, saving is enabled at a frequency of savecnfreq. Lines relevant to saving properties and visulization are discussed in later part of this manual.
How to record properties
Settings in the ctrl file
At the bottom of .ctrl file, following lines are needed,
#Save Properties saveprop = 1 savepropfreq = 1
which indicate saving properties is enabled and the saving frequency is once per simulation step.
Description of recorded properties
By default, properties are saved to the following path
~/Codes/ParaDiS/outputs/ctrl_file_name/properties
Usually there are six files.
1 time_Plastic_strain contains two columns of data:
1: elapsed simulation time 2: plastic strain
2 stress_Plastic_strain is generated only if loadType = 1 (constant strain rate) or 4 (cyclic loading). The first two columns of this file are plastic strain and stress respectively. 3 stress_Total_strain is generated only if loadType = 1. The first two columns of this file are total strain and stress. Notice that total strain is the sum of plastic strain and elastic strain. 4 density The first three columns of this file are
1: plastic strain 2: total strain 3: dislocation density (/m^2)
5 alleps contrains 9 columns of data:
1: smulation timestep number 2: elapsed simulation time 3-8: plastic strain component 9: dislocation density
6 epsdot contains 2 columns of data
1: elapsed simulation time 2: plastic strain rate
For detailed descriptions of the recorded data, please refer to the official ParaDiS manual.
Gnuplot
Gnuplot is a useful plotting software. You can plot the obtained properties data with gnupot, for example, stress vs strain curve. Before using gnuplot, make sure it is installed on your cluster. You also need to enable X server (test by typing xclock). By typing gnuplot, you can start using following command lines. For a given (X,Y) data pair, the gnuplot command is
gnuplot> plot "filename" using 1:2 with lines
where "using 1:2" means plotting using the 1st and 2nd column from the given data file. "with lines" is a style option, you can change to "with points", "with steps", "with dots" or "with impulses". Click [1] for a detailed description of the styles. To draw multiple lines, you can type
gnuplot> plot "A" using 1:2 title "data A" with lines, \ "B" u 1:2 t "data B" with p, \ "C" u 2:3 t "data C" w d
You can use \ to separate continuous lines. title is used to label different lines. You can abbreviate some of the commands, for example, u for using, t for title, w for with, l for lines, p for points and d for dots.
To change the labels of X and Y axes, you can use
gnuplot> set xlabel "x-axis" gnuplot> set ylabel "y-xais" gnuplot> set title "y vs x" gnuplot> replot
You can also change the ranges of X and Y axes,
gnuplot> set xrange [0:2] gnuplot> set yrange [0:1] gnuplot> replot
To assign the color of the line(line type), you need to add corresponding number after "l". For example, black solid(-1), black dot(0), red(1), green (2), blue(3), margenta(4), cyan(5) and yellow(6). To modify the width of the line at the same time, you can use option "w l lt 2 lw 4", for instance, where lt means linetype and lw means linewidth.
How to visualize the results
To visualize the simulation results, here are two methods. One is to run simulation interactively one one CPU and perform observation. The other is to record the configuration at certain simulation steps and make movies from them.
xwindow
Make sure x11 server is enabled for the session(test by "xclock"; using ssh -X username@domain when you log into clusters) and the ParaDiS code is compiled for single CPU("MODE = SERIAL, XLIB_MODE = ON" in makefile.setup). Check the following setting in /inputs/paradis.xdefaults if you set it as "winDefaultsFile" in your ctrl file.
enable_window = 1
By executing
bin/paradis -d runs/datafile.data runs/ctrlfile.ctrl
You should be able to see the xwindow. By setting "maxstep = 0" in the ctrl file, you can obtain the configuration (snapshot) for any given simulation step. When the simulation step is large, you might want to consider the other method.
Atomeye
Atomeye is a powerful visualization tool. To enable Atomeye, you need to add following lines in the ctrl file
atomeye = 1 atomeyefreq = 1 atomeyesegradius = 5
To view atomeye files, you can use command
$ tools/atomeye/A.i686 outputs/filename/atomeye/paradis0001.cfg
To create jpg files press y to convert all cfg files in the same directory.
To convert these jpg files to a movie, the command is
$ convert -delay 4 -quality 95 *.jpg mymovie.mpg
where -delay and -quality flags are optional; if -delay > 4, extra frames are added to make the movie run more slowly; default quality is 75 and the maximum is 100. Note that encode packages need to be preinstalled for the machine or cluster.
How to submit parallel jobs
pbs file
In a typical pbs file, we have following lines as header:
#!/bin/bash #PBS -N jobname #PBS -j oe #PBS -l nodes=8:ppn=8,walltime=24:00:00 #PBS -V ### --------------------------------------- ### BEGINNING OF EXECUTION ### --------------------------------------- echo The master node of this job is `hostname` echo The working directory is `echo $PBS_O_WORKDIR` echo This job runs on the following nodes: echo `cat $PBS_NODEFILE` ncpu=`cat $PBS_NODEFILE | wc -w` echo "Number of processors = $ncpu " ### end of information preamble cd $PBS_O_WORKDIR echo $PWD #cmd="mpiexec -np $ncpu bin/meam-lammps_mpich scripts/work/si_ge/si-meam-lammps.tcl" #time $cmd
You need to set the job name, wait time, number of nodes and ppn (number of processors per node, for example, ppn = 8 for wcr clusters)
To submit jobs using a single processor, you need to add the following line
#bin/paradis -d runs/datafile.data runs/ctrlfile.ctrl &> serial.out
where log file is serial.out.
For parallel jobs, add
#mpiexec -np $ncpu bin/paradis -d runs/datafile.data runs/ctrlfile.ctrl &> parallel.out
where log file is parallel.out.
What if the job is stopped
You can continue from the stored files under directory outputs/filename/restart by
$bin/paradis outputs/filename/restart/restart.cn