m (→VASP on SU-AHPCRC) |
m (→Computing Bulk Modulus of Au) |
Revision as of 13:47, 14 November 2008
In this document, we describe how to compile VASP program on several computer clusters in the Mechanical Engineering Department of Stanford University. These include mc-cc, wcr, and su-ahpcrc, all of which are linux clusters. We then give examples of how to use VASP to compute the bulk modulus of Au and ZrO2.
Contents |
VASP on MC-CC
In general we follow the instructions given in http://cms.mpi.univie.ac.at/vasp/vasp/node16.html
First, go to vasp.4.lib/ directory. Copy the vasp.4.lib/makefile.mc-cc file to Makefile and then compile. This can be done by the following commands.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.lib_makefile.mc-cc.txt -O Makefile make clean make
This will create libdmy.a in this directory.
Next, go to vasp.4.6/ directory. Copy the vasp.4.6/makefile.mc-cc file to Makefile and then compile. This can be done by the following commands.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.6_makefile.mc-cc.txt -O Makefile make clean make
This will create executable vasp in this directory.
Notice that in both makefiles, we use the /opt/mpich/intel/bin/mpif90 compiler. Different clusters have different mpi compilers and they have different speeds. Intel compilers usually perform better than generic compilers on intel linux clusters. Make sure in your directories, you do not have another file named makefile, which takes precedence over Makefile.
The binary (executable) file vasp can run in both serial mode (e.g. ./vasp) and parallel mode (e.g. mpiexec -np 4 vasp in a PBS script). The following table compares the time to run a simple benchmark case (one Au atom, LDA, ENCUT=400, ISMEAR=1, SIGMA=0.1, KPOINTS=21x21x21) using our executable here and the one available at /share/apps/vasp.4.6/bin/vasp. Our executable is about 70% faster.
| Number of CPUs | vasp compiled here | /share/apps/vasp.4.6/bin/vasp |
|---|---|---|
| 1 | 68 (seconds) | 116 (seconds) |
| 2 | 50 (seconds) | 86 (seconds) |
| 4 | 56 (seconds) | (cannot run -- killed) |
VASP on WCR
The procecure is similar to that on MC-CC, except that different compilers need to be used.
First, the command mpi-selector allows us to choose among different MPI compilers installed on the cluster.
$ mpi-selector --list mvapich_gcc-0.9.9 mvapich_gcc-1.0 mvapich_intel-0.9.9 mvapich_intel-1.0 mvapich_pgi-0.9.9 mvapich_pgi-1.0 openmpi_gcc-1.2.2 openmpi_intel-1.2.2 openmpi_pgi-1.2.2
This gives us a list of choices. Next, we choose mvapich_intel-0.9.9 by
$ mpi-selector --set mvapich_intel-0.9.9
You can double-check that your choice has been made by
$ mpi-selector --query default:mvapich_intel-0.9.9 level:user
Now you need to log-out of the cluster. When you log-in again, all MPI library paths will be correctly set up for you, e.g.
$ which mpif90 /usr/mpi/intel/mvapich-0.9.9/bin/mpif90
Now we go to the vasp.4.lib/ directory and execute the following commands.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.lib_makefile.wcr.txt -O Makefile make clean make
This will create libdmy.a in this directory.
Next, go to vasp.4.6/ directory and execute the following commands.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.6_makefile.wcr.txt -O Makefile make clean make
This will create executable vasp in this directory. This time the executable vasp can not run interactively, but can only run in the queue through a PBS script (e.g. mpiexec -np 4 vasp). Make sure in your directories, you do not have another file named makefile, which takes precedence over Makefile.
There is another executable at /share/apps/vasp.4.6/vasp, which can execute in both serial and parallel mode. To see that this executable contains MPI functions, use command
nm /share/apps/vasp.4.6/vasp | grep -i "MPI"
The following table compares the time to run the same benchmark case as above using both executables. Our executable shows speed up with multiple CPUs.
| Number of CPUs | vasp compiled here | /share/apps/vasp.4.6/vasp | vasp another build (below) |
|---|---|---|---|
| 1 | 76 (seconds) | 75 (seconds) | 48 (seconds) |
| 2 | 59 (seconds) | 72 (seconds) | 34 (seconds) |
| 4 | 35 (seconds) | 64 (seconds) | 29 (seconds) |
| 8 | 37 (seconds) | 65 (seconds) | 31 (seconds) |
Another way to compile vasp on WCR is the following. First
$ mpi-selector --unset
Log out WCR and log in again. Go to vasp.4.lib/.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.lib_makefile.wcr-intel.txt -O Makefile make clean make
This will re-create the libdmy.a in this directory. Next, go to vasp.4.6/.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.6_makefile.wcr-intel.txt -O Makefile make clean make
This will create executable vasp in this directory. The performance of this executable is listed in the fourth column of the table above.
VASP on SU-AHPCRC
The procecure is similar to that on WCR. First, we choose the mvapich_intel-0.9.9 compiler by
$ mpi-selector --set mvapich_intel-0.9.9
Now you need to log-out of the cluster and log-in again to have the MPI library paths correctly set up. Next Go to the vasp.4.lib/ directory and execute the following commands.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.lib_makefile.su-ahpcrc.txt -O Makefile make clean make
This will create libdmy.a in this directory.
Next, go to vasp.4.6/ directory and execute the following commands.
rm makefile wget http://micro.stanford.edu/mediawiki-1.11.0/images/Vasp.4.6_makefile.su-ahpcrc.txt -O Makefile make clean make
This will create executable vasp in this directory. This time the executable vasp can not run interactively, but can only run in the queue through a PBS script (e.g. mpiexec --comm=ib -np 4 vasp). Notice that here we need to specify the communication channel. (For mvapich1 we need to use --comm=ib and for mvapich2 use --comm=pmi.)
The following table provides the timing information for the same benchmark case studied above.
| Number of CPUs | vasp compiled here |
|---|---|
| 1 | 32 (seconds) |
| 2 | 23 (seconds) |
| 4 | 19 (seconds) |
| 6 | 28 (seconds) |
| 8 | 28 (seconds) |
| 16 | 203 (seconds) |
A Word of Caution: Make sure to run a few test cases to confirm your executable not only runs but produces the correct numerical results. For example, we have found that on su-ahpcrc, function BRMIX (broyden.f) was giving serious errors. This was solved by changing the compilation options to "OFLAG=-O1 -mtune core2 -axW -unroll".
Computing Bulk Modulus of Au
In the following, we give an example of how to use VASP to compute the bulk modulus of LDA-Au. We performed this calculation on MC-CC in serial model in the ~/Codes/VASP/runs/Au/LDA/perfect.21x21x21 directory. This directory contains the following files.
INCAR
ENCUT = 400 ISMEAR = 1 SIGMA = 0.1
KPOINTS
21x21x21 0 0 = automatic generation of k-points Monkhorst 21 21 21 0 0 0
POSCAR
POSCAR for FCC Au (created manually) 4.068 0 0.5 0.5 0.5 0 0.5 0.5 0.5 0 1 Cartesian (real coordinates r) 0 0 0
To do this calculation, you also need to put the LDA pseudopotential file as POTCAR in this directory.
Now we are ready to run
vasp
To compute the equilibrium lattice constant, cohesive energy and bulk modulus, we use the following script auto.B.serial to run vasp repeated with different lattice constants.
#!/bin/bash rm WAVECAR for a in 4.056 4.058 4.060 4.062 4.064 4.066 4.068 do cat > POSCAR << FIN POSCAR for FCC Au (created manually) $a 0 0.5 0.5 0.5 0 0.5 0.5 0.5 0 1 Cartesian (real coordinates r) 0 0 0 FIN echo "a=$a" ./vasp E=`tail -1 OSZICAR` echo $a $E | sed -s 's/F=//; s/E0=//; s/d E =//;' >> Elatt.B.dat p=`grep pressure OUTCAR | cut -b 25-34` echo $a $p >> platt.B.dat done
After running it as ./auto.B.serial, it will create data files Elatt.B.dat and platt.B.dat.
Launch octave and run the following functions fit_a0EB.m and fit_a0B.m,
fit_a0EB('Elatt.B.dat');
fit_a0B ('platt.B.dat');
The first line fits the energy data to a quadratic curve and computes the equilibrium lattice constant, cohesive energy and bulk modulus. The second line fits the pressure data to a linear curve and computes the equilibrium lattice constant and bulk modulus. In this example, the result is a0 = 4.60 angstrom, Ecoh = -4.39 eV, B = 190 GPa.
To run vasp in parallel, you need to submit vasp.pbs as
qsub vasp.pbs
You will need the following two files to do this calculation in parallel on SU-AHPCRC:
auto.B.par and
B.pbs.
Computing Bulk Modulus of ZrO2
In the following, we give an example of how to use VASP to compute the bulk modulus of GGA-ZrO2. We performed this calculation on SU-AHPCRC in parallel mode (8 CPUs). The running directory contains the following files.
INCAR
ZrO2 calculation ISMEAR = 0 ISIF = 2 PREC = High
KPOINTS
6x6x6 0 0 = automatic generation of k-points Monkhorst 6 6 6 0 0 0
B.pbs
#!/bin/bash #PBS -N ZrO2_bulk #PBS -j oe #PBS -l nodes=1:ppn=8,walltime=6:00:00 #PBS -V ### --------------------------------------- ### BEGINNING OF EXECUTION ### --------------------------------------- echo The master node of this job is `hostname` echo The working directory is `echo $PBS_O_WORKDIR` echo This job runs on the following nodes: echo `cat $PBS_NODEFILE` ncpu=`cat $PBS_NODEFILE | wc -w` echo "Number of processors = $ncpu " ### end of information preamble cd $PBS_O_WORKDIR echo $PWD ./auto.B.par $ncpu
auto.B.par
#!/bin/bash ncpu=$1 rm WAVECAR for a in 5.130 5.135 5.140 5.145 5.150 5.155 do cat > POSCAR << FIN POSCAR for ZrO2 (created by auto.B.par) $a 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00 4 8 Direct 0 0 0 0 0.5 0.5 0.5 0 0.5 0.5 0.5 0 0.25 0.25 0.25 0.25 0.25 0.75 0.25 0.75 0.25 0.25 0.75 0.75 0.75 0.25 0.25 0.75 0.25 0.75 0.75 0.75 0.25 0.75 0.75 0.75 FIN echo "a=$a ncpu=$ncpu" mpiexec --comm=ib -np $ncpu ../../../bin/vasp E=`tail -1 OSZICAR` echo $a $E | sed -s 's/F=//; s/E0=//; s/d E =//;' >> Elatt.B.dat p=`grep pressure OUTCAR | cut -b 25-34` echo $a $p >> platt.B.dat done
You also need to put the pseudopotential file (for Zr and O together) as POTCAR in this directory. Submit the calculation by
qsub B.pbs
The job will create data files Elatt.B.dat and platt.B.dat. Similar to the case of Au, we can compute the bulk modulus from these data by fitting the energy - lattice constant relation to a quadratic curve. In this example, the result is a0 = 5.146 angstrom, Ecoh = -9.47 eV, B = 239 GPa. You will need to make n=12; and N=12; if you want to use Matlab function fit_a0EB.m. Allowing the Zr and O sub-lattices to relax relative to each other doesn't seem to change the results. Note that in reality the cubic zirconia phase studied here is unstable (against the tetragonal phase). The cubic phase is only stablized if Yttrium dopants are added, forming yttria-stablized-zirconia (YSZ). (See http://www.springerlink.com/content/g72mw270608t757t/fulltext.pdf)