In order to properly intercept termination signals, pbslam must be exec'ed, replacing the shell which invokes it. pbslam checks for this, and will complain if it is not in the proper location within the PBS process hierarchy.
For added flexibility, pbslam provides two different strategies for mapping processes onto processors; these are described in detail in the section on Process Mapping. Which strategy is best depends on the requirements of the application, the number and type of nodes requested for the job, and the number of processes which will be run on those nodes.
pbslam supports two different schemes for mapping LAM processes onto PBS virtual processors. We call one of these schemes the "VP order", and the other "node order". By default, pbslam uses VP order; node order is invoked with the -n option. The contents of PBS_NODEFILE, as well as the mapping of processes onto nodes, is displayed on stdout when the -v option is specified.
VP Order: Processes are assigned one per VP in the order listed in PBS_NODEFILE. (Note that there may be more than one VP per node.) If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, then wrap around to the beginning of the VP list, assigning an additional process to each VP. This procedure repeats until all processes have been assigned.
Node Order: Processes are assigned one per node, wrapping around until all of the VP slots on all of the nodes are filled. If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, wrap around and assign an additional process to each node (rather than VP), repeating until all processes have been assigned.
Example 1: Our first example illustrates the difference in assignment strategies for a job which maps 12 processes onto 8 virtual processors which are spread across four nodes. Assume the following PBS job request:
qsub -l nodes=2:single+1:dual:ppn=2+1:quad:ppn=4
#!/usr/bin/csh
exec pbslam -c 12 myprog
^DOn SciClone, the resulting PBS_NODEFILE might look like:
ty01
ty02
tn01
tn01
hu01
hu01
hu01
hu01With the default VP order, processes would be mapped as follows:
p0 -> ty01
p1 -> ty02
p2 -> tn01
p3 -> tn01
p4 -> hu01
p5 -> hu01
p6 -> hu01
p7 -> hu01
p8 -> ty01
p9 -> ty02
p10 -> tn01
p11 -> tn01 If the -n option had been used on the pbslam command, the mapping would instead be:
p0 -> ty01
p1 -> ty02
p2 -> tn01
p3 -> hu01
p4 -> tn01
p5 -> hu01
p6 -> hu01
p7 -> hu01
p8 -> ty01
p9 -> ty02
p10 -> tn01
p11 -> hu01 Example 2: The -n option is particularly useful if you want exclusive use of a multi-processor node, but only want to use a subset of the processors on each node. The following example allocates all 16 processors on 8 dual-cpu nodes (thereby ensuring that the job has exclusive use of the nodes), but only assigns one process to each node:
qsub -l nodes=8:compute:dual:ppn=2
#!/usr/local/bin/tcsh
exec pbslam -n -c 8 myprog
^DExample 3: To route communication over Myrinet instead of Ethernet, use the -M option in conjunction with the "myri" node property:
qsub -l nodes=16:dual:myri:ppn=2
#!/usr/local/bin/tcsh
exec pbslam -M myprog
^DCircumstances may arise in which pbslam (or any other PBS job, for that matter) might not be able to find and kill all of the processes belonging to it. If a pristine execution environment is essential, additional checks (beyond -C or -X) may be needed to ensure that no stray processes reside on a node.