Running Hybrid MPI/OpenMP Simulations - Revision history

Amin1: /* References */

2015-11-07T14:47:09Z

References

← Older revision		Revision as of 14:47, 7 November 2015
Line 22:		Line 22:

	==References==		==References==
	https://www.cct.lsu.edu/~estrabd/intro-hybrid-mpi-openmp.pdf		*https://www.cct.lsu.edu/~estrabd/intro-hybrid-mpi-openmp.pdf
	https://wikis.nyu.edu/display/NYUHPC/Running+jobs+-+MPI		*https://wikis.nyu.edu/display/NYUHPC/Running+jobs+-+MPI
	https://www.mir.wustl.edu/Portals/0/Documents/Uploads/CHPC/WashU_7_mvapich.pdf		*https://www.mir.wustl.edu/Portals/0/Documents/Uploads/CHPC/WashU_7_mvapich.pdf

Amin1: /* OpenMP setting */

2015-11-04T22:08:43Z

OpenMP setting

← Older revision		Revision as of 22:08, 4 November 2015
Line 11:		Line 11:

	==OpenMP setting==		==OpenMP setting==
	One of the crucial settings that we need be careful about is the "processor affinity". By default, multi-threading affinity is not set in mvapich2. That means all your OpenMP threading will be bound to a single processor if you have ~~not~~ ~~changed~~ ~~this~~ ~~setting~~ ~~(and~~ if ~~you have compiled it through~~ mpicc, mpic++ or mpif90). This is because mvapich2 gives priority to MPI processors and does not allow OpenMP to use more than one CPU (even if there are available cpus for OpenMP threads). This setting can be changed through MV2_ENABLE_AFFINITY variable. In your terminal or PBS file type in the following command to enable OpenMP threads to run on different cores:		One of the crucial settings that we need be careful about is the "processor affinity". By default, multi-threading affinity is not set in mvapich2. That means, by default, all your OpenMP threading will be bound to a single processor (if you have compiled it through mvapich2 wrappers such as mpicc, mpic++ or mpif90). This is because mvapich2 gives priority to MPI processors and does not allow OpenMP to use more than one CPU (even if there are available cpus for OpenMP threads). This setting can be changed through MV2_ENABLE_AFFINITY variable. In your terminal or PBS file type in the following command to enable OpenMP threads to run on different cores:

	export MV2_ENABLE_AFFINITY=0		export MV2_ENABLE_AFFINITY=0

Amin1: /* OpenMP setting */

2015-11-04T22:06:02Z

OpenMP setting

← Older revision		Revision as of 22:06, 4 November 2015
Line 11:		Line 11:

	==OpenMP setting==		==OpenMP setting==
	One of the ~~main~~ settings that we need be careful about is the "processor affinity". By default, multi-threading affinity is not set in mvapich2. That means all your OpenMP threading will be bound to a single processor if you have not changed this setting (and if you have compiled it through mpicc, mpic++ or mpif90). This is because mvapich2 gives priority to MPI processors and does not allow OpenMP to use more than one CPU (even if there are available cpus for OpenMP threads). This setting can be changed through MV2_ENABLE_AFFINITY variable. In your terminal or PBS file type in the following command to enable OpenMP threads to run on different cores:		One of the crucial settings that we need be careful about is the "processor affinity". By default, multi-threading affinity is not set in mvapich2. That means all your OpenMP threading will be bound to a single processor if you have not changed this setting (and if you have compiled it through mpicc, mpic++ or mpif90). This is because mvapich2 gives priority to MPI processors and does not allow OpenMP to use more than one CPU (even if there are available cpus for OpenMP threads). This setting can be changed through MV2_ENABLE_AFFINITY variable. In your terminal or PBS file type in the following command to enable OpenMP threads to run on different cores:

	export MV2_ENABLE_AFFINITY=0		export MV2_ENABLE_AFFINITY=0

Amin1: /* Running Hybrid MPI/OpenMP Simulations */

2015-11-04T22:05:17Z

Running Hybrid MPI/OpenMP Simulations

← Older revision		Revision as of 22:05, 4 November 2015
Line 1:		Line 1:
	=Running Hybrid MPI/OpenMP Simulations=		=Running Hybrid MPI/OpenMP Simulations=
	This tutorial explains how to run a multi-threaded hybrid MPI/OpenMP simulation using mvapich2 library. mvapich2 is currently installed on both MC2 and WCR clusters. In this page we assume the code is already written in the hybrid fashion, e.g. both MPI and OpenMP headers are included in the code and you are able to compile and run your code ~~using either MPI or OpenMP~~.		This tutorial explains how to run a multi-threaded hybrid MPI/OpenMP simulation using mvapich2 library. mvapich2 is currently installed on both MC2 and WCR clusters. In this page we assume the code is already written in the hybrid fashion, e.g. both MPI and OpenMP headers are included in the code and you are able to compile and run your code.

	==Assigning the number of CPUs==		==Assigning the number of CPUs==

Amin1: /* Assigning the number of CPUs */

2015-11-04T06:21:35Z

Assigning the number of CPUs

← Older revision		Revision as of 06:21, 4 November 2015
Line 4:		Line 4:
	==Assigning the number of CPUs==		==Assigning the number of CPUs==
	Suppose you have 16 CPUs and want to have two MPI processors, each with 8 OpenMP threads.		Suppose you have 16 CPUs and want to have two MPI processors, each with 8 OpenMP threads.
	*Assigning OpenMP threads: ~~export~~		*Assigning OpenMP threads:
	export OMP_NUM_THREADS=8		export OMP_NUM_THREADS=8
	*Running:		*Running:

Amin1: /* Assigning the number of CPUs */

2015-11-04T06:21:09Z

Assigning the number of CPUs

← Older revision		Revision as of 06:21, 4 November 2015
Line 5:		Line 5:
	Suppose you have 16 CPUs and want to have two MPI processors, each with 8 OpenMP threads.		Suppose you have 16 CPUs and want to have two MPI processors, each with 8 OpenMP threads.
	*Assigning OpenMP threads: export		*Assigning OpenMP threads: export
	OMP_NUM_THREADS=8		export OMP_NUM_THREADS=8
	*Running:		*Running:
	mpirun -np 2 a.out		mpirun -np 2 a.out

Amin1: /* OpenMP setting */

2015-11-04T06:20:32Z

OpenMP setting

← Older revision		Revision as of 06:20, 4 November 2015
Line 11:		Line 11:

	==OpenMP setting==		==OpenMP setting==
	One of the main settings that we need be careful about is the "processor affinity". By default, multi-threading affinity is not set in mvapich2. That means all your OpenMP threading will be bound to a single processor if you have not changed this setting (and if you have compiled it through mpicc, mpic++ or mpif90). This is because mvapich2 gives priority to MPI ~~threads~~ and does not allow OpenMP to use more than one CPU (even if there are available cpus for OpenMP threads). This setting can be changed through MV2_ENABLE_AFFINITY variable. In your terminal or PBS file type in the following command to enable OpenMP threads to run on different cores:		One of the main settings that we need be careful about is the "processor affinity". By default, multi-threading affinity is not set in mvapich2. That means all your OpenMP threading will be bound to a single processor if you have not changed this setting (and if you have compiled it through mpicc, mpic++ or mpif90). This is because mvapich2 gives priority to MPI processors and does not allow OpenMP to use more than one CPU (even if there are available cpus for OpenMP threads). This setting can be changed through MV2_ENABLE_AFFINITY variable. In your terminal or PBS file type in the following command to enable OpenMP threads to run on different cores:

	export MV2_ENABLE_AFFINITY=0		export MV2_ENABLE_AFFINITY=0

Amin1: /* References */

2015-11-04T06:19:26Z

References

← Older revision		Revision as of 06:19, 4 November 2015
Line 24:		Line 24:
	https://www.cct.lsu.edu/~estrabd/intro-hybrid-mpi-openmp.pdf		https://www.cct.lsu.edu/~estrabd/intro-hybrid-mpi-openmp.pdf
	https://wikis.nyu.edu/display/NYUHPC/Running+jobs+-+MPI		https://wikis.nyu.edu/display/NYUHPC/Running+jobs+-+MPI
			https://www.mir.wustl.edu/Portals/0/Documents/Uploads/CHPC/WashU_7_mvapich.pdf

Amin1: /* Running Hybrid MPI/OpenMP Simulations */

2015-11-04T06:15:24Z

Running Hybrid MPI/OpenMP Simulations

← Older revision		Revision as of 06:15, 4 November 2015
Line 20:		Line 20:
	sed -n "1~${PBS_NUM_PPN}p" $PBS_NODEFILE > my_nodefile		sed -n "1~${PBS_NUM_PPN}p" $PBS_NODEFILE > my_nodefile
	mpirun -f my_nodefile -np $PBS_NUM_NODES my_program		mpirun -f my_nodefile -np $PBS_NUM_NODES my_program

			==References==
			https://www.cct.lsu.edu/~estrabd/intro-hybrid-mpi-openmp.pdf
			https://wikis.nyu.edu/display/NYUHPC/Running+jobs+-+MPI

Amin1: /* Binding MPI runs to different nodes */

2015-11-04T06:14:07Z

Binding MPI runs to different nodes

← Older revision		Revision as of 06:14, 4 November 2015
Line 16:		Line 16:

	==Binding MPI runs to different nodes==		==Binding MPI runs to different nodes==
	By default, mvapich2 tries to put all MPI ~~threads~~ on adjacent cores (not nodes). For example if you request 2 nodes and 8 cores through your PBS file and use "mpirun -np 2 a.out" both MPI processors will be assigned to the first node and the second node will be empty. In order to reserve the cores for OpenMP and put MPI processors on different nodes, you must use a modified nodefile as following:		By default, mvapich2 tries to put all MPI processors on adjacent cores (not nodes). For example if you request 2 nodes and 8 cores through your PBS file and use "mpirun -np 2 a.out" both MPI processors will be assigned to the first node and the second node will be empty. In order to reserve the cores for OpenMP and put MPI processors on different nodes, you must use a modified nodefile as following:

	sed -n "1~${PBS_NUM_PPN}p" $PBS_NODEFILE > my_nodefile		sed -n "1~${PBS_NUM_PPN}p" $PBS_NODEFILE > my_nodefile