For added flexibility, pbsmpich provides two different strategies for mapping processes onto processors; these are described in detail in the section on Process Mapping. Which strategy is best depends on the requirements of the application, the number and type of nodes requested for the job, and the number of processes which will be run on those nodes.
pbsmpich supports two different schemes for mapping MPICH processes onto PBS virtual processors. We call one of these schemes the "VP order", and the other "node order". By default, pbsmpich uses VP order; node order is invoked with the -n option. The contents of PBS_NODEFILE, as well as the mapping of processes onto nodes, is displayed on stdout when the -v option is specified.
VP Order: Processes are assigned one per VP in the order listed in PBS_NODEFILE. (Note that there may be more than one VP per node.) If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, then wrap around to the beginning of the VP list, assigning an additional process to each VP. This procedure repeats until all processes have been assigned.
Node Order: Processes are assigned one per node, wrapping around until all of the VP slots on all of the nodes are filled. If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, wrap around and assign an additional process to each node (rather than VP), repeating until all processes have been assigned.
Example 1: Our first example illustrates the difference in assignment strategies for a job which maps 12 processes onto 8 virtual processors which are spread across four nodes. Assume the following PBS job request:
qsub -l nodes=2:single+1:dual:ppn=2+1:quad:ppn=4
#!/usr/bin/csh
pbsmpich -c 12 myprog
^DOn SciClone, the resulting PBS_NODEFILE might look like:
ty01
ty02
tn01
tn01
hu01
hu01
hu01
hu01With the default VP order, processes would be mapped as follows:
p0 -> ty01
p1 -> ty02
p2 -> tn01
p3 -> tn01
p4 -> hu01
p5 -> hu01
p6 -> hu01
p7 -> hu01
p8 -> ty01
p9 -> ty02
p10 -> tn01
p11 -> tn01 If the -n option had been used on the pbsmpich command, the mapping would instead be:
p0 -> ty01
p1 -> ty02
p2 -> tn01
p3 -> hu01
p4 -> tn01
p5 -> hu01
p6 -> hu01
p7 -> hu01
p8 -> ty01
p9 -> ty02
p10 -> tn01
p11 -> hu01 Example 2: The -n option is particularly useful if you want exclusive use of a multi-processor node, but only want to use a subset of the processors on each node. The following example allocates all 16 processors on 8 dual-cpu nodes (thereby ensuring that the job has exclusive use of the nodes), but only assigns one process to each node:
qsub -l nodes=8:compute:dual:ppn=2
#!/usr/local/bin/tcsh
pbsmpich -n -c 8 myprog
^DExample 3: To route communication over Myrinet instead of Ethernet, use the -M option in conjunction with the "myri" node property:
qsub -l nodes=16:dual:myri:ppn=2
#!/usr/local/bin/tcsh
pbsmpich -M myprog
^DExample 4: In contrast with pbslam, pbsmpich doesn't have to be exec'ed, which means that a series of programs can be run from a single job:
qsub -l nodes=8:typhoon
#!/usr/local/bin/tcsh
cd ~/results
pbsmpich ~/bin/preprocess args... ‹raw_data ›data
pbsmpich ~/bin/compute args... ‹data ›output
pbsmpich ~/bin/analyze args... ‹output ›final
^DCircumstances may arise in which pbsmpich (or any other PBS job, for that matter) might not be able to find and kill all of the processes belonging to it. If a pristine execution environment is essential, additional checks (beyond -C or -X) may be needed to ensure that no stray processes reside on a node.