Micro and Nano Mechanics Group

(written by Alfredo Correa, with contributions of Wei Cai)

In this document, we describe how to compile QBOX program on wcr.stanford.edu (Linux parallel cluster) and other systems.

I designed this wiki such that you could in principle copy and paste the code part in your terminal. Custom files are linked from this wiki and can be downloaded from the command line with 'wget'. I encourage to do this to other writers of the group wiki. In any case, be aware of error messages, please report if something it is not working from this instructions or edit this wiki for other obvious typos.


General Remarks

Based on the Qbox build instructions page we will give some additional details. The difficult part of the installation is to correctly build the dependence libraries. In particular

  1. BLAS and Lapack http://math-atlas.sourceforge.net and http://www.netlib.org/lapack/
  2. BLACS http://www.netlib.org/blacs
  3. ScaLAPACK http://www.netlib.org/scalapack
  4. FFTW 2.1.5 http://www.fftw.org
  5. Apache Xerces-C http://xerces.apache.org/xerces-c

these libraries will have to be installed in your user space directory. To achieve this I use to create a directory called ~/usr where libraries and header files will stay:

 mkdir -p $HOME/usr
 mkdir -p $HOME/usr/local

I also use to download software packages to ~/soft. This will make easier to establish absolute compilation paths for BLACS and Scalapack which have a non-standard installation procedure.

 mkdir $HOME/soft

This document does not cover the usage of Qbox but just the compilation, a manual of Qbox is provided in the Qbox home page.

First step is to choose a compiler and stick to it for compiling the different libraries and the final Qbox executable. In systems where gcc-4 and gcc-3 coexist this can be challenging since different forms of invoking a compiler can switch from one compiler set to the other. For example in RedHat, f77 calls gcc-3 while f90 or gfortran calls gcc-4. The issue is worst when calling the MPI compiler wrappers. In any case run the compiler with the -v option to make sure the same version of gcc is used across all the steps. (A common mistake is to mix g77 and gfortran. They actually belong to different versions of gcc.)

Install a C++ and Fortran compiler*

Most likely C++ is installed in your system, if not try to follow this. For example, in Ubuntu just do

sudo apt-get install g++ gfortran

Otherwise see this other tutorial 'Install GCC'.

Install MPICH2*

cd ~/soft
wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/$MPI_VER/mpich2-$MPI_VER.tar.gz
tar -zxvf mpich2-$MPI_VER.tar.gz
cd mpich2-$MPI_VER
./configure --prefix=$HOME/usr --exec-prefix=$HOME/usr`uname -m` --enable-f90 --enable-sharedlibs=gcc
time make --jobs=1
make install

(20 minutes) If you are going to use this just installed version of MPI, make sure to specify $(HOME)/usr/bin/mpicxx as your C++ compiler in the *.mk file.

(The library can be uninstalled with the command $EPREFIX/sbin/mpeuninstall.) (The option --enable-sharedlibs does not work in the Cygwin enviroment)

or from SVN

(http://wiki.mcs.anl.gov/mpich2/index.php/Getting_And_Building_MPICH2, http://www.cmiss.org/openCMISS/wiki/SettingUpAndUsingMPICH2)

mkdir -p ~/soft/mpich2.svn
cd ~/soft/mpich2.svn
svn co https://svn.mcs.anl.gov/repos/mpi/mpich2/trunk .
./configure --prefix=$HOME/usr --exec-prefix=$HOME/usr`uname -m` --enable-cxx --enable-shared
time make --jobs=1
make install

(8 minutes) This instalation of MPI is not optimal in a arbitrary cluster. Most clusters will have their own super optimised implementation over special communication methods (ethernet, infiniband, etc).

You do a very basic test by running:

mpiexec -n 2 echo 'a'

Install Blas and Lapack*

*Most systems have Blas and Lapack installed already, so you may want to skip this section. For example, they are installed system-wide already. Note: Atlas does not compile with Fortran 77 compilers (use Fortran 95 compilers).

(Requirement: a fortran compiler, e.g. apt-get install gfortran)

Manual installation of tunned BLAS (ATLAS)

Skip this section if you want to install the non-optimized version of BLAS that comes with Lapack.

To get BLAS (ATLAS implementation) (http://www.netlib.org or http://math-atlas.sourceforge.net):

cd ~/soft
rm -rf atlas$ATLAS_VER.tar.bz2 ATLAS
wget http://downloads.sourceforge.net/project/math-atlas/Stable/$ATLAS_VER/atlas$ATLAS_VER.tar.bz2
tar -jxvf atlas$ATLAS_VER.tar.bz2

Apparently the library doesn't do the correct tunning if the CPU clocking is enabled (usually in laptop and desktops), this line may disable the throttling

sudo /usr/bin/cpufreq-selector -g performance

(If you are not root in the system you can force compilation anyway --who knows what the performance will be, it can be very bad-- by adding the configure option -Si cputhrchk 0)

The library must be built in a different directory than the sources.

mkdir ATLAS/build
cd ATLAS/build
../configure --incdir=$HOME/usr/include --libdir=$HOME/usr`uname -m`/lib 
#-b 64

If we want to use Intel compilers, use

../configure --incdir=$HOME/usr/include --libdir=$HOME/usr`uname -m`/lib -b 64 -C if ifort -F if -O -C ic icc -F ic -O

The compilation takes ~15 minutes. You can copy/link the newly generated files libf77blas.a libatlas.a to your desired location (for example ~/usr/lib) or just

make install

To have them in the location defined by prefix.

Note that on the current version of ATLAS contains bugs when compiled on 64-bit machines using Intel compilers. This causes codes to crash during run time. In this case (e.g. on mc2), we must not use ATLAS but need to use the internal BLAS package contained in Lapack (see below).


You can then get Lapack (http://www.netlib.org/lapack) from the home page, download the file and untar it:

cd ~/soft
wget http://www.netlib.org/lapack/lapack.tgz
tar -zxvf lapack.tgz
cd lapack-*
cp make.inc.example make.inc

There are two options here.

Option 1. If you installed ATLAS (above) then you have to:

First modify make.inc to allow linking with the (just compiled) ATLAS, for example.

BLASLIB = $(HOME)/usr/lib/libf77blas.a $(HOME)/usr/lib/libatlas.a

Modify the following lines if you want to use the intel compiler

FORTRAN  = ifort
LOADER   = ifort

Then do

make --jobs=1 #~15 minutes

Note that on 64-bit intel machines (e.g. mc2) our experience has been that the ATLAS package contains bugs (see above). So we must use the following option.

Option 2. Lapack includes a basic (non optimized) version of BLAS

make blaslib
make --jobs=2 #~7 minutes

Create a symlink to the newly created library (there is no make install):

mkdir -p ~/usr/lib
cd ~/usr/lib
ln -sf ~/soft/lapack-3.2.2/lapack_LINUX.a liblapack.a
ln -sf ~/soft/lapack-3.2.2/blas_LINUX.a libblas.a #if any

Install BLACS

Download and untar BLACS from BLACS papers page:

cd ~/soft
wget http://www.netlib.org/blacs/mpiblacs.tgz
tar -zxvf mpiblacs.tgz

We have to have our own version of a file called Bmake.inc in the ~/soft/BLACS directory. The following file will serve for BLACS/Bmake.inc

wget http://micro.stanford.edu/mediawiki/images/8/8f/BLACS_Bmake.inc.txt -O Bmake.inc

The file is just the same as BMAKES/Bmake.MPI-LINUX with the following changes in the variables definition:

< BTOPdir = $(HOME)/soft/BLACS
< MPIdir = /opt/mpich/gnu

otherwise use

sed 's:BTOPdir = $(HOME)/BLACS:BTOPdir = $(HOME)/soft/BLACS:' BMAKES/Bmake.MPI-LINUX  | sed 's:MPIdir = /usr/local/mpich:MPIdir = $(HOME)/usr:' > Bmake.inc

If you have a local installation of MPI libraries, you need to modify that line to MPIdir = $(HOME)/usr.

In some platforms (notably Ubuntu --if you are doing a full manual installation--) you have to modify Bmake.inc to have the following line:


In some other platforms you need a different setting (I don't understand the logic of this). To know what should be used you must (compile and) run a little program.

make -C INSTALL xintface

in principle this should give the right option, although in some cases I had to use a different one.

which will tell you what option to use. Do the modification to Bmake.inc if necessary. This is an example of Bmake.inc for a manual installation in Linux.

Having this setting correctly is very important, if not the compilation of this (and other libraries) will actually finish without any warnings, except that at some point when we need not compile a program we will get errors about 'undefined references'. Also remember the option used because it will have to be consistent in the compilation of Scalapack (see below).

Also you may need to change the compiler name

F77   = gfortran


F77   = ifort

instead of g77.

Then change to the source directory and make, then copy the compiled libraries to ~/usr/local/lib/

make #<1 minute

mkdir -p ~/usr/lib$CLUSTER
cd ~/usr/lib$CLUSTER
cp ~/soft/BLACS/LIB/blacsCinit_MPI-LINUX-0.a libblacsCinit-mpi.a
cp ~/soft/BLACS/LIB/blacsF77init_MPI-LINUX-0.a libblacsF77init-mpi.a                    
cp ~/soft/BLACS/LIB/blacs_MPI-LINUX-0.a libblacs-mpi.a

Install Scalapack

Now it is the turn for Scalapack, get it from the Scalapack home page, untar it:

cd ~/soft
wget http://www.netlib.org/scalapack/scalapack-1.8.0.tgz
tar -zxvf scalapack-1.8.0.tgz
cd scalapack-1.8.0

and put this compilation input file scalapack/SLmake.inc in the directory

wget http://micro.stanford.edu/mediawiki/images/8/8d/Scalapack_SLmake.inc.txt -O SLmake.inc

which is just the same as 'SLmake.inc.example' with the following changes:

< home          = $(HOME)/soft/scalapack-1.8.0
< F77           = /opt/mpich/gnu/bin/mpif77
< CC            = /opt/mpich/gnu/bin/mpicc


sed 's:$(HOME):$(HOME)/soft:' SLmake.inc.example > SLmake.inc

Again, for a local installation of MPI use $(HOME)/usr/bin instead of /opt/mpich/gnu/bin. In the line CDEF it should appear the same option as in the INTFACE option for the compilation of BLACS, for example -DAdd_. This is an SLmake.inc example for a manual installation in Linux.

Then you are ready to compile:

make --jobs=1

(4 minutes).

On mc2, use make, do not use make --jobs=8, which leads to missing packages.

Link the resulting library to ~/usr/lib (there is no make install):

mkdir --parents ~/usr/lib$CLUSTER
cd ~/usr/lib$CLUSTER
cp ~/soft/scalapack-1.8.0/libscalapack.a .

Precompiled packages

If you have a Linux system (or any system listed here, here or here) it might be easier to just download and install a ready to use file, no compilation required:

mkdir --parents ~/soft/blas_LINUX
cd ~/soft/blas_LINUX
wget http://www.netlib.org/blas/archives/blas_linux.tgz
tar -zxvf blas_linux.tgz
mkdir -p ~/usr/local/blas/lib
cd ~/usr/local/blas/lib
ln -s ~/soft/blas_LINUX/blas_linux.a libblas.a
mkdir -p ~/soft/atlas_LINUX
cd ~/soft/atlas_LINUX
wget http://www.netlib.org/atlas/archives/linux/atlas3.6.0_Linux_P4SSE2.tgz
tar -zxvf atlas3.6.0_Linux_P4SSE2.tgz
mkdir -p ~/usr/local/atlas/lib
cd ~/usr/local/atlas/lib
ln -s ~/soft/atlas_LINUX/Linux_P4SSE2/lib/*.a .
mkdir -p ~/soft/lapack_LINUX
cd ~/soft/lapack_LINUX
wget http://www.netlib.org/lapack/archives/lapack_linux.tgz
tar -zxvf lapack_linux.tgz
mkdir -p ~/usr/local/lapack/lib
cd ~/usr/local/lapack/lib
ln -s ~/soft/lapack_LINUX/lapack_linux.a liblapack.a
mkdir -p ~/soft/blacs_LINUX
cd ~/soft/blacs_LINUX
wget http://www.netlib.org/blacs/archives/blacs_MPI-LINUX-0.tgz
tar -zxvf blacs_MPI-LINUX-0.tgz
mkdir -p ~/usr/local/blacs/lib
cd ~/usr/local/blacs/lib
ln -s ~/soft/blacs_LINUX/blacs_MPI-LINUX-0.a libmpiblacs.a
ln -s ~/soft/blacs_LINUX/blacsCinit_MPI-LINUX-0.a libmpiblacsCinit.a
ln -s ~/soft/blacs_LINUX/blacsF77init_MPI-LINUX-0.a libmpiblacsF77init.a
mkdir -p ~/soft/scalapack_LINUX
cd ~/soft/scalapack_LINUX
wget http://www.netlib.org/scalapack/archives/scalapack_LINUX.tgz
tar -zxvf scalapack_LINUX.tgz
mkdir -p ~/usr/local/scalapack/lib
cd ~/usr/local/scalapack/lib
ln -s ~/soft/scalapack_LINUX/scalapack_LINUX.a libscalapack.a

Install FFTW

Among the possible FFT libraries that can be linked to Qbox, the easiest library to install is FFTW 2.1.5 (FFTW3 is available but Qbox uses only FFTW2). Simply download it from the FFTW download page and decompress it:

mkdir -p ~/soft
cd ~/soft
wget http://www.fftw.org/fftw-2.1.5.tar.gz
tar -zxvf fftw-2.1.5.tar.gz
cd fftw-2.1.5

Then do the usual configure and make install procedure, indicating that we want a private installation in our ~/usr directory

mkdir -p ~/usr
./configure --prefix=$HOME/usr --libdir=$HOME/usr/lib$CLUSTER --enable-shared=yes --disable-fortran --with-pic
make -j 1
make install

This will install the FFTW2 libraries in ~/usr/include and ~/usr/lib.

Note that it is NOT necessary to install the MPI version of fftw, Qbox does not make use of it. If for some reason you need to install the MPI version of FFTW use the option --enable-mpi MPICC=$HOME/usr/bin/mpicc in the ./configure command line.

(see also Install FFTW3)

Install Xerces

Now the Xerces XML library, get it from the Xerces download page and untar it, also define an enviroment variable were the source of xerces is located:

cd ~/soft
wget http://mirror.cloudera.com/apache//xerces/c/2/sources/xerces-c-src_2_8_0.tar.gz
tar -zxvf xerces-c-src_2_8_0.tar.gz
cd xerces-c-src_2_8_0
cd src/xercesc

then run this script and the usual make install

make clean
./runConfigure -plinux -cgcc -xg++ -minmem -nsocket -tnative -rpthread -P $HOME/usr -s -C --libdir=$HOME/usr/lib$CLUSTER
time make --jobs=1
make install

(5 minutes) This will make static xerces libraries installed in ~/usr/lib$CLUSTER and ~/usr/include/xerces. To have both the static and shared libraries recompile again without the -s option.

To uninstall:

rm -rf ~/usr/include/xercesc/* ~/usr/libhera/libxerces-*

Xerces 3

Warning: I found problems (pseudopotentials not loading) using Xerces 3 with (at least) Qbox v1.45 and v1.50. Try with care.

Xerces version 3 was released, it has much easier installation procedure (./configure --prefix=$HOME/usr; make install;) which takes ~10 minutes (1 minute with make -j 8).

cd ~/soft
wget http://apache.spinellicreations.com//xerces/c/3/sources/xerces-c-$XERCES_VER.tar.gz
tar -zxvf xerces-c-$XERCES_VER.tar.gz
cd xerces-c-$XERCES_VER
./configure --prefix=$HOME/usr --libdir=$HOME/usr/lib$CLUSTER --with-pic --enable-transcoder-iconv
NUM_CORES=`cat /proc/cpuinfo | grep processor | wc -l`
make --jobs=$NUM_CORES install 

(Some flags could be important to compile in BG/Q)

in Qbox 1.50 the flag -DXERCESC_3_0_1 has to be active in your target .mk file to use this version of xerces.

Qbox 1.45, instead, has to be slightly modified in order to run:

In the file WavefunctionHandler.C it says XMLString::release(&b);, that should be modified to


and where is says XMLByte* b = Base64::decode((XMLByte*)content.c_str(), &length); cast to (XMLSize_t*) before &length

Also the final executable has to be linked to additional libraries, such as ICU libraries (-lsicuuc -lsicudata in Ubuntu) or -lcurl (in RedHat).

Ready to compile Qbox

Having the previous libraries installed in our own directory ~/usr and the rest of them already installed by the system administrator we shouldn't have any problems compiling Qbox

First step is to get the source code from Qbox download page and decompress it.

#1.45.0 = old version
cd ~/soft
wget http://fpmd.ucdavis.edu/software/qbox/qbox-$QBOX_VER.tgz
tar -zxvf qbox-$QBOX_VER.tgz
cd qbox-$QBOX_VER/src 

We have to have a Makefile input file for our own system (wcr.stanford.edu) in the current directory; for that purpose we have created qbox-1.45.0/src/wcr.mk, we will also need to specify our target system (according to the filename wcr.mk). (Although the original tutorial was made for compilation in wcr.stanford.edu, all the steps seem to work for mc-cc.stanford.edu as well with no modification.) Note that this compilation uses MPI library that corresponds to /opt/mpich/gnu/bin/mpicxx.

 wget http://micro.stanford.edu/mediawiki-1.11.0/images/Qbox_wcr.mk.txt -O wcr.mk
 export TARGET=wcr

wcr.mk file distributed here is based on x8664_gcc.mk with the following changes:

 <  MPIDIR=/opt/mpich/gnu
 <  FFTWDIR=$(HOME)/usr
 <  BLASDIR=$(HOME)/usr/local/lib
 <  LAPACKDIR=$(HOME)/usr/local/lib
 <  CXX=/opt/mpich/gnu/bin/mpicxx
 <          -llapack -lf77blas -latlas -lm  \
 <          -Xlinker -Bstatic \
 <           -lc -static-libgcc -lmpich  -lxerces-c \
 <  # gfortran
 <  BLACSdir      = $(HOME)/usr/local/lib
 <  SCALAPACK_DIR = $(HOME)/usr/local/lib

Alternatively this other modified version x8664_gcc.mk was used to manually compile under Linux. Now we are ready to make

make | tee make.log

This can take a couple of minutes and generate an executable named 'qb' . We can check that the qb is properly compiler by doing

ldd ./qb

and see whether all necessary libraries can be found in the system. Afterwards we can try to run a small example,

 cd ~/soft/qbox-1.45.0/examples/ch4
 /opt/mpich/gnu/bin/mpirun -np 2 ~/soft/qbox-1.45.0/src/qb gs.i

If it works we can copy the executable to our bin directory. Since the qbox versions change rather often it is convenient to keep track of the executable version and make a link to the latest version available.

 cp ~/soft/qbox-1.45.0/src ~/usr/local/bin/qbox-1.45.0
 ln -s ~/usr/local/bin/qbox-1.45.0 ~/usr/local/bin/qbox

Since the libraries are statically linked 'qb' executable does not depend in any actual file in our /usr/lib.

Fast compilation

Compilation of Qbox is relatively quick, few minutes at most. However if you are modifying the code frequently and compiling from scratch it can be useful to compile the code taking advantage of multi-processor computers by means of a feature of GNU make. For example you can do:

 time make -j 8

in an 8 processor machine and expect much better compilation times than with the usual 'time make'.

Quick compilation in Ubuntu

It is possible to compile Qbox using the libraries and development files provided by the Ubuntu repositories.

In Ubuntu 9.04

sudo aptitude install \
 libblas-dev liblapack-dev \
 blacsgf-mpich-dev scalapack-mpich-dev \
 fftw-dev fftw3-dev \
 libxerces-c2-dev libmpich1.0dev \
cd ~/soft/qbox-1.45.0
wget http://micro.stanford.edu/mediawiki/images/3/38/Qbox_ubuntu-9.04.mk.txt -O ubuntu-9.04.mk
export TARGET=ubuntu-9.04

after successful compilation run a small example

 cd ../examples/ch4
 /usr/lib/mpich/bin/mpirun -np 2 ../../qb gs.i

Intel Based Compilation

In some test cases its been observed that compiling with the Intel compiler and MKL can speed up the calculations by a factor of 2.

Ready Makefiles-includes

After compiling Qbox in many platforms, keeping track of the many Makefile-includes (*.inc) used and its combinations can be difficult. For that reason I decided to pack the different Makefile-includes used, and incrementally add them to this section:

 export TARGET=x8664_gcc
 export TARGET=x8664_gcc_Ubuntu

Other tips

GCC 4 warnings and errors

GCC 4 produces many warnings like this,

 warning: deprecated conversion from string constant to ‘char*’

The ideal solution would be to make the code conform with the new standard (by replacing all offending instances of 'char*' by 'char const*'), but that's up to the developers.

Meanwhile we can disable that warning with the option '-Wno-write-strings' passed to the compiler. That can be added to the include-Makefile (*.mk) in the following way:

 CXXFLAGS=-g -O4 -D$(PLT) $(INCLUDE) $(PLTFLAGS) $(DFLAGS) -Wno-write-strings

another set of errors raise with gcc 4.4 (memset undefined) can be solved editing Context.h and adding


Finding unresolved functions

Quite often the source files are compiled into objective (*.o) files but the final link (ld) step is unsuccessful, with error messages such as "__some_function__ is unresolved". At this point, the nm command may be useful. It allows us to see whats are implemented and what are required (from another .o file) in a given .o file.

For example, go to the Qbox src/ directory and try

nm Matrix.o

This will print out a lot of lines. Each line tells us about a function that is either implemented in this object file, or is required by but not implemented in this object file. For example, you will see a line

U  zscal_

U means not implemented. This means that somewhere in Matrix.o there is a function call to zscal_ but this function is not implemented in Matrix.o. Hence Matrix.o needs to be linked to another .o (object), .a (library) or .so (dynamic library) file that implements the zscal_ function.

We know zscal_ is a BLAS function. So we can look into the BLAS library file, e.g.

nm /usr/lib64/libblas.a | grep zscal

We will see

T  zscal_

T means implemented. Hence the libblas.a library can provide the function needed by Matrix.o. Of course, a computer system usually have many libraries installed that provide the same (or similar) functions. So we need to be careful in choosing the libraries to link (e.g. by specifying library paths using the -L option).

One thing that the nm command can show us is whether the function name in the .o file (from the source code) and that in the library file matches --- pay attention to the final underscore "_". In this case, it tells us that we need to turn on the -DAdd_ option in the Qbox makefile (for SU-AHPCRC).

Once we realize that there are some undefined (U) symbol, it is our job to find which is the file we need to link to (for example with the -l option). We can Google it or search our local system for such symbol. I find this command line very useful to find where a symbol is defined:

for a in /usr/lib/*.a; do echo $a; nm -o $a | grep FUNCTION_NAME; done

This allows to search in many files at once. Always look for the function name without the underscore in order to capture which is the correct function name underscoring (e.g. zscal_ vs zscal) defined in the system.

In some systems compiled functions are stored in shared object files `.so` in ELF format. To find functions in these files use this program and option `readelf -Ws`; for example `readelf -Ws /usr/lib674/openmpi/lib/libscalapack.so`.

Compilation in Cygwin

Cygwin provides Lapack, BLAS and Xerces. These can be installed with the usual setup.exe or with apt-cyg.

BLACS, Scalapack, fftw2 can be compiled in the usual way (see above). One thing to remember (in addition to the previous instructions) is that part of the dll libraries (with the cyg* prefix, e.g. cyglapack-0.dll) are located at $HOME/usr/bin, /usr/bin and /usr/lib/lapack, so these directories must be in the PATH variable (yes, the windows PATH variable within Cygwin). For example,

export PATH=/usr/lib/lapack:$HOME/usr/bin:$PATH

Similar Guides

https://www.alcf.anl.gov/user-guides/bgp-qbox http://fpmd.ucdavis.edu/qbox-list/

(end of compilation tutorial)