SciClone Cluster Project Computational Science Cluster
Home
Introduction
Sponsors
Research
Hardware
Software
User Info
Documentation

Name: LAM/MPI

Version: 7.1.2

Description:

MPI message-passing libraries for distributed and shared memory environments, plus the ROMIO MPI I/O package. Includes native language interfaces for C, C++, and FORTRAN. LAM implements several different mechanisms (RPI's) for local communication within SMP nodes and remote communication between nodes. These include:

  • tcp - TCP/IP over Ethernet, Myrinet, or InfiniBand for local and remote communication.
  • sysv - TCP/IP for remote communication with System V semaphores for local communication.
  • usysv - TCP/IP for remote communication with back-off spin locks for local communication.
  • gm - GM over Myrinet for local and remote communication.
  • lamd - Local and remote communication via background LAM daemons.

GM support is new in LAM 7, and this option will normally deliver the best performance on SciClone's Myrinet-enabled nodes.

Previous versions of LAM required the choice of RPI to be made at compile time. In LAM 7, RPI's are selected at runtime.

LAM 7 is installed in both 32-bit (v8plusa, v8plusb, sse2a) and 64-bit (v9a, v9b, amd64a) versions for use on SciClone's UltraSPARC and Opteron processors.

Availability: UltraSPARC and x86

Installation Path:

/usr/local/[v8plusa,v8plusb,sse2a,v9a,v9b,amd64a]/[generic,ultra3cu,opteron]/lam-7.1.2

Documentation:

Usage Notes:

LAM/MPI programs on SciClone must be initiated via PBS or TORQUE using the pbslam command rather than mpirun. See the SciClone User's Guide and pbslam man page for complete details.

LAM doesn't provide a compiler wrapper script for Fortran 90 (mpif90). To compile Fortran 90 code with LAM's mpif77 compiler wrapper, it is necessary to set some additional flags to tell the compiler to behave like Fortran 90 instead of Fortran 77:

$LAMHOME/bin/mpif77 -f77=%none -ftrap=common ...

A total of six different variants of LAM are installed on SciClone to provide for:

  • 32-bit vs. 64-bit addressing (v8*,sse2a vs. v9*,amd64a)
  • UltraSPARC II, UltraSPARC III, and Opteron processors (v*a, v*b, opteron)

"v8plusa" and "v9a" will run satisfactorily on any UltraSPARC node in SciClone, but "v8plusb" and "v9b" should only be used on nodes with UltraSPARC III processors (c5, c7, c7a, n2, db1, and s3 node types).

The "v8plusa" and "v9a" versions are built for use with a "generic" UltraSPARC CPU, while the "v8plusb" and "v9b" versions are compiled to take advantage of the "ultra3cu" chip found only on our UltraSPARC III nodes.

The "sse2a" and "amd64a" versions are for use on Opteron-based nodes (c8, s4, s4a node types).

The choice between "generic", "ultra3cu", and "opteron" is determined automatically if you use the default .cshrc file described below.

The choice of transport layer is a little trickier. "tcp" uses TCP sockets for all communication, both local and remote, while "sysv" and "usysv" use shared memory to optimize local communication within a node. "usysv" uses a combination of spinlocks and semaphores to coordinate access to shared memory. "sysv" relies only on semaphores. If only one process is spawned on each node, then all three transports should provide similar performance. If multiple processes are spawned on a node, then some experimentation is in order to determine which transport will deliver the best performance for a given application.

On Myrinet-enabled nodes (c2, c4, c5, c6, c6a, c7, c7a, n1, n2, db1 node types), the "gm" transport will usually provide the best performance, often by a wide margin.

MPICH-GM typically provides slightly better performance than LAM's "gm" RPI over Myrinet, but the convenience of selecting a communication mechanism at run-time rather than compile-time may make LAM the better choice for applications which need the flexibility to execute in many of SciClone's different subclusters.

LAM's "ib" (InfiniBand) RPI is not available on SciClone because current versions of Solaris do not support the Mellanox Verbs API (VAPI) which LAM requires. For native InfiniBand communication, use MVAPICH instead.

While the "tcp", "sysv", and "usysv" RPI's can be used for TCP/IP communication across Myrinet or InfiniBand, performance will be relatively poor compared to the "gm" RPI (Myrinet) or MVAPICH (InfiniBand).

By default, pbslam will choose the best available RPI based on the set of nodes allocated to a job. The user can override this with the "-r" option.

Environment Variables

For proper operation, the LAMHOME environment variable must be set in the user's ~/.cshrc file to specify the desired variant of LAM. $LAMHOME/bin should be in the user's search path, and $LAMHOME/man should be included in the MANPATH. The file /usr/local/etc/templates/cshrc contains the current recommended environment settings and includes options for specifying which version of LAM to use, and for switching between LAM and other MPI packages.