In this document, we describe how to compile QBOX program on wcr.stanford.edu (Linux parallel cluster).
I designed this wiki such that you could in principle copy and paste the code part in your terminal. Custom files are linked from this wiki and can be downloaded from the command line with 'wget'. I encourage to do this to other writers of the group wiki. In any case, be aware of error messages, please report if something it is not working from this instructions or edit this wiki for other obvious typos.
Contents |
General Remarks
Based on the Qbox build instructions page we will give some additional details. The difficult part of the installation is to correctly build the dependence libraries. In particular
- BLAS and Lapack
- BLACS http://www.netlib.org/blacs
- ScaLAPACK http://www.netlib.org/scalapack
- FFTW 2.1.5 http://www.fftw.org
- Apache Xerces-C http://xerces.apache.org/xerces-c
these libraries will have to be installed in your user space directory. To achieve this I use to create a directory called ~/usr where libraries and header files will stay:
mkdir -p $HOME/usr/local/lib
I also use to download software packages to ~/soft. This will make easier to establish absolute compilation paths for BLACS and Scalapack which have a non-standard installation procedure.
mkdir $HOME/soft
This document does not cover the usage of Qbox but just the compilation, a manual of Qbox is provided in the Qbox home page.
Install Blas and Lapack*
*Most systems have Blas and Lapack installed already, so you may want to skip this section. For example, they are installed system-wide already.
To get BLAS (ATLAS implementation) (http://www.netlib.org or http://math-atlas.sourceforge.net):
cd ~/soft wget http://voxel.dl.sourceforge.net/sourceforge/math-atlas/atlas3.8.3.tar.gz tar -zxvf atlas3.8.3.tar.gz
The library must be built in a different directory than the sources.
mkdir ATLAS-build cd ATLAS-build ../ATLAS/configure --prefix=$HOME/usr make
The compilation takes ~15 minutes. You can copy the newly generated files libf77blas.a libatlas.a to your desired location (for example ~/usr/local/atlas/lib) or just
make install
To have them in the location defined by prefix.
You can then get Lapack (http://www.netlib.org/lapack) from the home page, download the file and untar it:
cd ~/soft wget http://www.netlib.org/lapack/lapack.tgz tar -zxvf lapack.tgz cd lapack-3.2 cp make.inc.example make.inc
First modify make.inc to allow linking with the (just compiled) ATLAS, for example.
BLASLIB = $(HOME)/usr/local/atlas/lib/libf77blas.a $(HOME)/usr/local/atlas/lib/libatlas.a
Lapack includes a basic (non optimized) version of BLAS, that can create confusion since we really need to link to the tuned version (above).
Then proceed to compile the rest of Lapack:
make
This will take another ~15 minutes.
Install BLACS
Next download and untar BLACS from BLACS papers page:
cd ~/soft wget http://www.netlib.org/blacs/mpiblacs.tgz tar -zxvf mpiblacs.tgz cd BLACS
We have to have our own version of a file called Bmake.inc in the ~/soft/BLACS directory. The following file will serve for BLACS/Bmake.inc
wget http://micro.stanford.edu/mediawiki-1.11.0/images/BLACS_Bmake.inc.txt -O Bmake.inc
The file is just the same as BMAKES/Bmake.MPI-LINUX with the following changes in the variables definition:
< BTOPdir = $(HOME)/soft/BLACS < MPIdir = /opt/mpich/gnu
If you have a local installation of MPI libraries, you need to modify that line to MPIdir = $(HOME)/usr.
In some platforms (notably Ubuntu --if you are doing a full manual installation--) you have to modify Bmake.inc to have the following line:
INTFACE = -DAdd_
In some other platforms you need a different setting (I don't understand the logic of this). To know what should be used you must (compile and) run a little program.
make -C INSTALL xintface INSTALL/EXE/xintface
which will tell you what option to use. Do the modification to Bmake.inc if necessary. This is an example of Bmake.inc for a manual installation in Linux.
Having this setting correctly is very important, if not the compilation of this (and other libraries) will actually finish without any warnings, except that at some point when we need not compile a program we will get errors about 'undefined references'. Also remember the option used because it will have to be consistent in the compilation of Scalapack (see below).
Then change to the source directory and make, then copy the compiled libraries to ~/usr/local/lib/
cd SRC/MPI make mkdir ~/usr/local mkdir ~/usr/local/lib cp ~/soft/BLACS/LIB/*.a ~/usr/local/lib
You will have the files:
$ ls ~/usr/local/lib/ blacsCinit_MPI-LINUX-0.a blacsF77init_MPI-LINUX-0.a blacs_MPI-LINUX-0.a
Install Scalapack
Now it is the turn for Scalapack, get it from the Scalapack home page, untar it:
cd ~/soft wget http://www.netlib.org/scalapack/scalapack-1.8.0.tgz tar -zxvf scalapack-1.8.0.tgz cd scalapack-1.8.0
and put this compilation input file scalapack/SLmake.inc in the directory
wget http://micro.stanford.edu/mediawiki-1.11.0/images/Scalapack_SLmake.inc.txt -O SLmake.inc
which is just the same as 'SLmake.inc.example' with the following changes:
< home = $(HOME)/soft/scalapack-1.8.0 < F77 = /opt/mpich/gnu/bin/mpif77 < CC = /opt/mpich/gnu/bin/mpicc
Again, for a local installation of MPI use $(HOME)/usr/bin instead of /opt/mpich/gnu/bin. In the line CDEF it should appear the same option as in the INTFACE option for the compilation of BLACS, for example -DAdd_. This is an SLmake.inc example for a manual installation in Linux.
Then you are ready to compile (4 minutes):
make
and copy the resulting library to ~/usr/local/lib
cp ~/soft/scalapack-1.8.0/libscalapack.a ~/usr/local/lib
Install FFTW
Among the possible FFT libraries that can be linked to Qbox, the easiest library to install is FFTW 2.1.5 (FFTW3 is available but Qbox uses only FFTW2). Simply download it from the FFTW download page and decompress it:
cd ~/soft wget http://www.fftw.org/fftw-2.1.5.tar.gz tar -zxvf fftw-2.1.5.tar.gz cd fftw-2.1.5
Then do the usual configure and make install procedure, indicating that we want a private installation in our ~/usr directory
./configure --prefix=$HOME/usr make make install
This will install the FFTW2 libraries in ~/usr/include and ~/usr/lib. Note that it is NOT necessary to install the mpi version of fftw, Qbox does not make use of it. If for some reason you need to install the MPI version of FFTW use the option --enable-mpi in the ./configure command line.
Install Xerces
Now the Xerces XML library, get it from the Xerces download page and untar it, also define an enviroment variable were the source of xerces is located:
cd ~/soft wget http://apache.siamwebhosting.com/xerces/c/2/sources/xerces-c-src_2_8_0.tar.gz tar -zxvf xerces-c-src_2_8_0.tar.gz cd xerces-c-src_2_8_0 export XERCESCROOT=$PWD cd src/xercesc
then run this script and the usual make install
./runConfigure -plinux -cgcc -xg++ -minmem -nsocket -tnative -rpthread -P $HOME/usr -s make make install
This will make static xerces libraries installed in ~/usr/lib and ~/usr/include/xerces.
Update: Xerces version 3.0 was released, it has much more easy installation procedure (./configure --prefix=$HOME/usr; make install;) which takes ~10 minutes. But Qbox has to be slightly modified in order to run:
In the file WavefunctionHandler.C it says XMLString::release(&b);, that should be modified to
XMLString::release((char**)&b);
Also the final executable has to be linked to ICU libraries (-lsicuuc -lsicudata).
Ready to compile Qbox
Having the previous libraries installed in our own directory ~/usr and the rest of them already installed by the system administrator we shouldn't have any problems compiling Qbox
First step is to get the source code from Qbox download page and decompress it.
cd ~/soft wget http://fpmd.ucdavis.edu/software/qbox/qbox-1.45.0.tgz tar -zxvf qbox-1.45.0.tgz cd qbox-1.45.0/src
We have to have a Makefile input file for our own system (wcr.stanford.edu) in the current directory; for that purpose we have created qbox-1.45.0/src/wcr.mk, we will also need to specify our target system (according to the filename wcr.mk). (Although the original tutorial was made for compilation in wcr.stanford.edu, all the steps seem to work for mc-cc.stanford.edu as well with no modification.) Note that this compilation uses MPI library that corresponds to /opt/mpich/gnu/bin/mpicxx.
wget http://micro.stanford.edu/mediawiki-1.11.0/images/Qbox_wcr.mk.txt -O wcr.mk export TARGET=wcr
wcr.mk file distributed here is based on x8664_gcc.mk with the following changes:
< MPIDIR=/opt/mpich/gnu < XERCESCDIR=$(HOME)/usr < FFTWDIR=$(HOME)/usr < BLASDIR=$(HOME)/usr/local/lib < LAPACKDIR=$(HOME)/usr/local/lib < CXX=/opt/mpich/gnu/bin/mpicxx < -llapack -lf77blas -latlas -lm \ < -Xlinker -Bstatic \ < -lc -static-libgcc -lmpich -lxerces-c \ < # gfortran < PLAT=LINUX < BLACSdir = $(HOME)/usr/local/lib < SCALAPACK_DIR = $(HOME)/usr/local/lib
Alternatively this other modified version x8664_gcc.mk was used to manually compile under Linux. Now we are ready to make
make
This can take a couple of minutes and generate an executable named 'qb' . We can check that the qb is properly compiler by doing
ldd ./qb
and see whether all necessary libraries can be found in the system. Afterwards we can try to run a small example,
cd ~/soft/qbox-1.45.0/examples/ch4 /opt/mpich/gnu/bin/mpirun -np 2 ~/soft/qbox-1.45.0/src/qb gs.i
If it works we can copy the executable to our bin directory. Since the qbox versions change rather often it is convenient to keep track of the executable version and make a link to the latest version available.
cp ~/soft/qbox-1.45.0/src ~/usr/local/bin/qbox-1.45.0 ln -s ~/usr/local/bin/qbox-1.45.0 ~/usr/local/bin/qbox
Since the libraries are statically linked 'qb' executable does not depend in any actual file in our /usr/lib.
Quick compilation in Ubuntu
It is possible to compile Qbox using the libraries and development files provided by Ubuntu. The downside is that it seems that it does not work with the amd64 version (either with the provided Scalapack or with a manually compiled one). In the 32bit version of Ubuntu just install the following packages from the standard repositories.
Install compilers and compiler libraries
sudo apt-get install cpp-4.2 g++-4.2 gcc-4.2 libstdc++6-4.2-dev sudo apt-get install gfortran-4.2
Install MPICH environment and intercommunication tools
sudo apt-get install libmpich1.0-dev libmpich1.0gf sudo apt-get install mpich-bin sudo apt-get install openssh-server ssh
Install numerical and parsing libraries
sudo apt-get install fftw-dev fftw2 sudo apt-get install libxerces-c2-dev libxerces-c28
Install parallel and serial linear algebra libraries
sudo apt-get install blacs1gf-mpich blacsgf-mpich-dev sudo apt-get install libblas3gf libgfortran3 sudo apt-get install liblapack3gf liblapack-dev sudo apt-get install scalapack-mpich-dev scalapack1-mpich
Now download the following prepared file qbox-1.45.0/src/Ubuntux86.mk and compile:
cd ~/soft/qbox-1.45.0 wget http://micro.stanford.edu/mediawiki-1.11.0/images/Qbox_ubuntux86.mk.txt -O Ubuntux86.mk export TARGET=Ubuntux86 make
after successful compilation run a small example
cd ../examples/ch4 /usr/lib/mpich/bin/mpirun -np 2 ../../qb gs.i
Notes: Qbox-<1.45 does not compile with g++-4.3 (this is the default compiler in Ubuntu 8.10), therefore g++-4.2 was explicitly specified in the provided makefile. Compilation warnings about 'char*' can be ignored. Some systems have MPI-LAM installed also, so make sure you use the mpich version of mpirun. If passwordless ssh is not enabled mpirun will request login password even for the localhost, to enable it follow the instruccion for passwordless ssh.
Other tips
Quite often the source files are compiled into objective (*.o) files but the final link (ld) step is unsuccessful, with error messages such as "__some_function__ is unresolved". At this point, the nm command may be useful. It allows us to see whats are implemented and what are required (from another .o file) in a given .o file.
For example, go to the Qbox src/ directory and try
nm Matrix.o
This will print out a lot of lines. Each line tells us about a function that is either implemented in this object file, or is required by but not implemented in this object file. For example, you will see a line
U zscal_
U means not implemented. This means that somewhere in Matrix.o there is a function call to zscal_ but this function is not implemented in Matrix.o. Hence Matrix.o needs to be linked to another .o (object), .a (library) or .so (dynamic library) file that implements the zscal_ function.
We know zscal_ is a BLAS function. So we can look into the BLAS library file, e.g.
nm /usr/lib64/libblas.a | grep zscal
We will see
T zscal_
T means implemented. Hence the libblas.a library can provide the function needed by Matrix.o. Of course, a computer system usually have many libraries installed that provide the same (or similar) functions. So we need to be careful in choosing the libraries to link (e.g. by specifying library paths using the -L option).
One thing that the nm command can show us is whether the function name in the .o file (from the source code) and that in the library file matches --- pay attention to the final underscore "_". In this case, it tells us that we need to turn on the -DAdd_ option in the Qbox makefile (for SU-AHPCRC).
Once we realize that there are some undefined (U) symbol, it is our job to find which is the file we need to link to (for example with the -l option). We can Google it or search our local system for such symbol. I find this command line very useful to find where a symbol is defined:
for a in /usr/lib/*.a; do echo $a; nm $a | grep FUNCTION_NAME; done
This allows to search in many files at once.
(end of compilation tutorial)