Parallel Cluster Guides

su-ahpcrc

The default settings for MPI will cause the code to use GCC rather than the Intel compilers. To change this:

• EXTREMELY IMPORTANT: Ensure that nothing related to Intel has been added to your path in .bashrc and .bash_profile
• Run:

$ mpi-selector --query
default:mvapich2_intel-1.2
level:user

• If your result is something other than the above (version number may differ), execute:

$ mpi-selector --set mvapich2_intel-1.2

• Log out and reconnect to the cluster
• If everything is correct, asking which mpicc should yield:

$ which mpicc
/usr/mpi/intel/mvapich2-1.2/bin/mpicc

• Add the following to makefile.sys, if not present:

########################################################
#
#    System type:  su-ahpcrc
#    
#    Stanford ME linux system using intel compilers
#
########################################################

#
#    Define parallel and serial compilers and compiler flags
#    and set the default compiler based on the execution mode
#    (defined by user in makefile)
#
CC_PARALLEL.su-ahpcrc       = mpicc
CPP_PARALLEL.su-ahpcrc      = mpicxx
CCFLAG_PARALLEL.su-ahpcrc   = -longdouble -DLONGDOUBLE -DPARALLEL=1 
CPPFLAG_PARALLEL.su-ahpcrc  = 

CC_SERIAL.su-ahpcrc         = icc
CPP_SERIAL.su-ahpcrc        = icpc
CCFLAG_SERIAL.su-ahpcrc     = -DLONGDOUBLE
CPPFLAG_SERIAL.su-ahpcrc    = 

F90.su-ahpcrc               = ifort
F90_OPTS.su-ahpcrc          = 
F90_LIB.su-ahpcrc           = -L/opt/intel/fce/10.1.015/lib -lifcore

CC.su-ahpcrc                = $(CC_$(MODE).su-ahpcrc)
CPP.su-ahpcrc               = $(CPP_$(MODE).su-ahpcrc)
CCFLAG.su-ahpcrc            = $(CCFLAG_$(MODE).su-ahpcrc)
CPPFLAG.su-ahpcrc           = $(CPPFLAG_$(MODE).su-ahpcrc)

XLIB_LIBDIR.su-ahpcrc       = /usr/X11R6/lib64
XLIB_LIB.su-ahpcrc          = -L$(XLIB_LIBDIR.su-ahpcrc) -lX11 -lpthread
XLIB_INCS.su-ahpcrc         =

MPI_LIBDIR.su-ahpcrc        = -L/export/apps/mvapich/intel/lib
MPI_LIB.su-ahpcrc           = -lmpich
MPI_INCS.su-ahpcrc          = 

OPENMP_FLAG.su-ahpcrc       = -openmp

#
#    Identify any additional libraries and paths needed for compilation
#    on this system type
#
LIB_PARALLEL.su-ahpcrc      =
INCS_PARALLEL.su-ahpcrc     =

LIB_SERIAL.su-ahpcrc        =-L../lib
INCS_SERIAL.su-ahpcrc       =

• When compiling the code, seeing icc and comments about vectorizing loops such as "StressTableGen.c(357): (col. 37) remark: LOOP WAS VECTORIZED." are signs that the Intel compilers were used.
• Use a PBS script similar to the following (/opt/mpiexec/bin/mpiexec --comm=pmi is the only important difference):

#!/bin/bash
#PBS -N ParaDiS
#PBS -j oe
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -V

### ---------------------------------------
### BEGINNING OF EXECUTION
### ---------------------------------------

echo The master node of this job is `hostname`
echo The working directory is `echo $PBS_O_WORKDIR`
echo This job runs on the following nodes:
echo `cat $PBS_NODEFILE`

ncpu=`cat $PBS_NODEFILE | wc -w`
echo "Number of processors = $ncpu "

### end of information preamble

cd $PBS_O_WORKDIR

echo $PWD

PARADIS_O_DIR="tests/fmm_8cpu_results"
mkdir -p $PARADIS_O_DIR
cmd="/opt/mpiexec/bin/mpiexec --comm=pmi -np $ncpu bin/paradis tests/fmm_8cpu.ctrl"
$cmd >& $PARADIS_O_DIR/paradis.log

• Submit the job using a command similar to:

$ qsub paradis.pbs