MPI message-passing libraries
for distributed and shared memory environments, plus the ROMIO
MPI I/O package. Includes native language interfaces for C, C++, and
FORTRAN. LAM implements several different mechanisms (RPI's) for local
communication within SMP nodes and remote communication between nodes.
These include:
- tcp
- TCP/IP over Ethernet, Myrinet, or InfiniBand for local and remote
communication.
- sysv
- TCP/IP for remote communication with System V semaphores for local
communication.
- usysv
- TCP/IP for remote communication with back-off spin locks for local
communication.
- gm -
GM over Myrinet for local and remote communication.
- lamd
- Local and remote communication via background LAM daemons.
GM support is new in
LAM 7, and this option will normally deliver the best performance on
SciClone's Myrinet-enabled nodes.
Previous versions of
LAM required the choice of RPI to be made at compile time. In LAM 7,
RPI's are selected at runtime.
LAM 7 is installed in
both 32-bit (v8plusa, v8plusb, sse2a) and 64-bit (v9a, v9b, amd64a)
versions for use on SciClone's UltraSPARC and Opteron processors.
LAM/MPI programs on SciClone
must be initiated via PBS or TORQUE using
the pbslam
command rather than mpirun. See the SciClone
User's Guide and pbslam
man page for complete details.
LAM doesn't provide a
compiler wrapper script for Fortran 90 (mpif90). To compile Fortran
90 code with LAM's mpif77 compiler wrapper, it is necessary to
set some additional flags to tell the compiler to behave like Fortran
90 instead of Fortran 77:
$LAMHOME/bin/mpif77
-f77=%none -ftrap=common ...
A total of six different
variants of LAM are installed on SciClone to provide for:
- 32-bit vs. 64-bit
addressing (v8*,sse2a vs. v9*,amd64a)
- UltraSPARC II, UltraSPARC
III, and Opteron processors (v*a, v*b, opteron)
"v8plusa" and
"v9a" will run satisfactorily on any UltraSPARC node in SciClone,
but "v8plusb" and "v9b" should only be used on nodes
with UltraSPARC III processors (c5, c7, c7a, n2, db1, and s3 node
types).
The "v8plusa"
and "v9a" versions are built for use with a "generic"
UltraSPARC CPU, while the "v8plusb" and "v9b" versions
are compiled to take advantage of the "ultra3cu" chip found
only on our UltraSPARC III nodes.
The "sse2a"
and "amd64a" versions are for use on Opteron-based nodes (c8,
s4, s4a node
types).
The choice between "generic",
"ultra3cu", and "opteron" is determined automatically
if you use the default .cshrc file described below.
The choice of transport
layer is a little trickier. "tcp" uses TCP sockets for all
communication, both local and remote, while "sysv" and "usysv"
use shared memory to optimize local communication within a node. "usysv"
uses a combination of spinlocks and semaphores to coordinate access
to shared memory. "sysv" relies only on semaphores. If only
one process is spawned on each node, then all three transports should
provide similar performance. If multiple processes are spawned on a
node, then some experimentation is in order to determine which transport
will deliver the best performance for a given application.
On Myrinet-enabled nodes
(c2, c4, c5, c6, c6a, c7, c7a, n1, n2, db1 node
types), the "gm" transport will usually provide the best
performance, often by a wide margin.
MPICH-GM
typically provides slightly better performance than LAM's "gm"
RPI over Myrinet, but the convenience of selecting a communication mechanism
at run-time rather than compile-time may make LAM the better choice
for applications which need the flexibility to execute in many of SciClone's
different subclusters.
LAM's "ib"
(InfiniBand) RPI is not available on SciClone because current versions
of Solaris do not support the Mellanox Verbs API (VAPI) which LAM requires.
For native InfiniBand communication, use MVAPICH instead.
While the "tcp",
"sysv", and "usysv" RPI's can be used for TCP/IP
communication across Myrinet or InfiniBand, performance will be relatively
poor compared to the "gm" RPI (Myrinet) or MVAPICH (InfiniBand).
By default, pbslam
will choose the best available RPI based on the set of nodes allocated
to a job. The user can override this with the "-r" option.
Environment
Variables
For proper operation,
the LAMHOME environment variable must be set in the user's ~/.cshrc
file to specify the desired variant of LAM. $LAMHOME/bin should be in
the user's search path, and $LAMHOME/man should be included in the MANPATH.
The file /usr/local/etc/templates/cshrc contains the current
recommended environment settings and includes options for specifying
which version of LAM to use, and for switching between LAM and other
MPI packages.
