In order to properly intercept termination signals, pbslam must be exec'ed, replacing the shell which invokes it. pbslam checks for this, and will complain if it is not in the proper location within the PBS process hierarchy.
For added flexibility, pbslam provides two different strategies for mapping processes onto processors; these are described in detail in the section on Process Mapping. Which strategy is best depends on the requirements of the application, the number and type of nodes requested for the job, and the number of processes which will be run on those nodes.
pbslam supports two different schemes for mapping LAM processes onto PBS virtual processors. We call one of these schemes "VP order", and the other "node order". By default, pbslam uses VP order; node order is invoked with the -n option. The contents of PBS_NODEFILE, as well as the mapping of processes onto nodes, is displayed on stdout when the -v option is specified.
VP Order: Processes are assigned one per VP in the order listed in PBS_NODEFILE. (Note that there may be more than one VP per node.) If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, then wrap around to the beginning of the VP list, assigning an additional process to each VP. This procedure repeats until all processes have been assigned.
Node Order: Processes are assigned one per node, wrapping around until all of the VP slots on all of the nodes are filled. If the number of processes requested by the -c option is larger than the number of VPs allocated to the job, wrap around and assign an additional process to each node (rather than VP), repeating until all processes have been assigned.
Example 1: Our first example illustrates the difference in assignment strategies for a job which maps 12 processes onto 8 virtual processors which are spread across four nodes. Assume the following PBS job request:
qsub -l nodes=2:single+1:dual:ppn=2+1:quad:ppn=4
#!/usr/bin/csh
exec pbslam -c 12 myprog
^DOn SciClone, the resulting PBS_NODEFILE might look like:
wh01
wh02
tw01
tw01
hu01
hu01
hu01
hu01With the default VP order, processes would be mapped as follows:
p0 -> wh01
p1 -> wh02
p2 -> tw01
p3 -> tw01
p4 -> hu01
p5 -> hu01
p6 -> hu01
p7 -> hu01
p8 -> wh01
p9 -> wh02
p10 -> tw01
p11 -> tw01 If the -n option had been used on the pbslam command, the mapping would instead be:
p0 -> wh01
p1 -> wh02
p2 -> tw01
p3 -> hu01
p4 -> tw01
p5 -> hu01
p6 -> hu01
p7 -> hu01
p8 -> wh01
p9 -> wh02
p10 -> tw01
p11 -> hu01 Example 2: The -n option is particularly useful if you want exclusive use of a multi-processor node, but only want to use a subset of the processors on each node. The following example allocates all 16 processors on 8 dual-cpu nodes (thereby ensuring that the job has exclusive use of the nodes), but only assigns one process to each node:
qsub -l nodes=8:compute:dual:ppn=2
#!/usr/local/bin/tcsh
exec pbslam -n -c 8 myprog
^DExample 3: To force both local and remote communication to use TCP/IP and to route remote traffic via InfiniBand rather than Ethernet, use the -N and -r options in conjunction with the "ib4x" node property:
qsub -l nodes=16:dual:ib4x:ppn=2
#!/usr/local/bin/tcsh
exec pbslam -N 8 -r tcp myprog
^DThe performance of Solaris 9 TCP/IP over Myrinet is especially poor with certain message sizes. Use of the tcp, sysv, usysv, and lamd RPI's in conjunction with Myrinet is therefore not recommended at this time.
Circumstances may arise in which pbslam (or any other PBS job, for that matter) might not be able to find and kill all of the processes belonging to it. If a pristine execution environment is essential, additional checks (beyond -C or -X) may be needed to ensure that no stray processes reside on a node.