MPICH-GM programs will only run on nodes with Myrinet interfaces. pbsmpichgm checks for this, and will complain if it determines that any of the nodes assigned to the job lack Myrinet connectivity. You can assure that only Myrinet nodes are assigned to a job by specifying the correct node properties with PBS's qsub command.
For added flexibility, pbsmpichgm provides two different strategies for mapping processes onto processors; these are described in detail in the section on Process Mapping. Which strategy is best depends on the requirements of the application, the number and type of nodes requested for the job, and the number of processes which will be run on those nodes.
MPI_Abort() function correctly,
so only processes which call MPI_Abort() directly will actually
exit, leaving other processes active. As a workaround, mpirun.ch_gm
can be configured to automatically kill all remaining processes in a job
at some specified time interval after the first process exits. pbsmpichgm
uses this feature to ensure that failed jobs are aborted in a timely manner
and that stray processes get cleaned up. By default, the delay is set
to 15 seconds. The -k option can be used to make this delay longer
or shorter, but values longer than 45 seconds may interfere with PBS's
job termination procedure and are therefore not recommended. Note that
the delay occurs even when all processes in a job exit normally and/or
at the same time, so large values for secs will slow down job termination
and tie up processors needlessly. If processes in an application are supposed
to exit at different times, there is a chance that slower processes could
be killed prematurely. To prevent this, insert an extra MPI_Barrier()
before the program's exit() or STOP statement
to ensure that all processes exit together. -l nodes=" option of qsub.
pbsmpichgm supports two different schemes for mapping MPI processes onto PBS virtual processors. We call one of these schemes "VP order", and the other "node order". By default, pbsmpichgm uses VP order; node order is invoked with the -n option. The contents of PBS_NODEFILE, as well as the mapping of processes onto nodes, is displayed on stdout when the -v option is specified.
VP Order: Processes are assigned one per VP in the order listed in PBS_NODEFILE. (Note that there may be more than one VP per node.) If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, then wrap around to the beginning of the VP list, assigning an additional process to each VP. This procedure repeats until all processes have been assigned.
Node Order: Processes are assigned one per node, wrapping around until all of the VP slots on all of the nodes are filled. If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, wrap around and assign an additional process to each node (rather than VP), repeating until all processes have been assigned.
Example 1: Our first example illustrates the difference in assignment strategies for a job which maps 12 processes onto 8 virtual processors which are spread across four nodes. Assume the following PBS job request:
qsub -l nodes=3:dual:myri:ppn=2+1:quad:myri:ppn=4
#!/usr/bin/csh
pbsmpichgm -c 12 -r hybrid myprog
^DOn SciClone, the resulting PBS_NODEFILE might look like:
tn01
tn01
tn02
tn02
tn03
tn03
hu01
hu01
hu01
hu01
With the default VP order, processes would be mapped as follows:
p0 -> tn01
p1 -> tn01
p2 -> tn02
p3 -> tn02
p4 -> tn03
p5 -> tn03
p6 -> hu01
p7 -> hu01
p8 -> hu01
p9 -> hu01
p10 -> tn01
p11 -> tn01 If the -n option had been used on the pbsmpichgm command, the mapping would instead be:
p0 -> tn01
p1 -> tn02
p2 -> tn03
p3 -> hu01
p4 -> tn01
p5 -> tn02
p6 -> tn03
p7 -> hu01
p8 -> hu01
p9 -> hu01
p10 -> tn01
p11 -> tn02Example 2: The -n option is particularly useful if you want exclusive use of a multi-processor node, but only want to use a subset of the processors on each node. The following example allocates all 16 processors on 8 dual-cpu nodes (thereby ensuring that the job has exclusive use of the nodes), but only assigns one process to each node:
qsub -l nodes=8:c2:myri:ppn=2
#!/usr/local/bin/tcsh
pbsmpichgm -n -c 8 myprog
^D-l nodes=" option of the qsub
command. See the SciClone User's
Guide for more information.
mpirun.ch_gm currently does not allow different executables to be started
on different nodes, and neither does pbsmpichgm.
Circumstances may arise in which pbsmpichgm (or any other PBS job, for that matter) might not be able to find and kill all of the processes belonging to it. If a pristine execution environment is essential, additional checks (beyond -C or -X) may be needed to ensure that no stray processes reside on a node.
MPI-Abort() does not operate correctly under MPICH-GM. See
the discussion of the -k option
for a workaround.