The GM communication system provides reliable, ordered delivery between communication endpoints, called "ports," with two levels of priority. This model is "connectionless" in that there is no need for client software to establish a connection with a remote port in order to communicate with it: the client software simply builds a message and sends it to any port in the network. (This apparently paradoxical "connectionless reliability" is achieved by GM maintaining reliable connections between each pair of hosts in the network and multiplexing the traffic between ports over these reliable connections.)
Under operating systems that provide memory protection, GM provides memory protected network access. It should be impossible for any non-privileged GM client application to use GM to access any memory other than the application's own memory, except as explicitly allowed by the GM API. The unforgeable source of each received message is available to the receiver, allowing the receiver to discard messages from untrusted sources.
The largest message GM can send or receive is limited to
bytes. However, because send and receive buffers must reside in DMAable
memory, the maximum message size is limited by the amount of DMAable
memory the GM driver is allowed to allocate by the operating system.
Most GM applications obtain DMAable memory using the straightforward
gm_dma_free() calls, but sophisticated
applications with large memory requirements may perform DMA memory
gm_deregister_memory() to pin and unpin memory on operating
systems that support memory registration.
Message order is preserved only for messages of the same priority, from the same sending port, and directed to the same receiving port. Messages with differing priority never block each other. Consequently, low priority messages may pass high priority messages, unlike in some other communication systems. Typical GM applications will either use only one GM priority, or use the high priority channel for control messages (such as client-to-client acks) or for single-hop message forwarding.
Both sends and receives in GM are regulated by implicit tokens,
representing space allocated to the client in various internal GM
queues, as depicted in the following figure. At initialization, the
client implicitly possesses
gm_num_send_tokens() send tokens, and
gm_num_receive_tokens() receive tokens. The client may call
certain functions only when possessing an implicit send or receive
token, and in calling that function, the client implicitly relinquishes
the token(1). The client
program is responsible for tracking the number of tokens of each type
that it possesses, and must not call any GM function requiring a token
when the client does not possess the appropriate token. Calling a GM API
function without the required tokens has undefined results, but GM
usually reports such errors, and such errors will not cause system
security to be violated.
As stated above, sends are token regulated. A client of a port may send
a message only when it possesses a send token for that port. By calling
a GM API send functions, the client implicitly relinquishes that send
token. The client passes a
context pointer to
the send function. When the send completes, GM calls
passing a pointer to the GM port, the client-supplied
pointer, and status code indicating if the send completed successfully
or with an error. When GM calls the client's
the send token is implicitly passed back to the client. Most GM programs,
which rely on GM's fault tolerance to handle transient network faults,
should consider a send completing with a status other than
GM_SUCCESS to be a fatal error. However, more sophisticated
programs may use the GM fault tolerance API extensions to handle such
non-transient errors. These extensions are described in an appendix.
It is important to note that the client-supplied
will be called only within a client's call to
the GM unknown event handler function that the client must call when
it receives an unrecognized event. The
is described in more detail below.
GM receives are also token regulated. After a port is opened, the
client implicitly possesses
tokens, allowing it to provide GM with up to this many receive buffers
gm_provide_receive_buffer(). With each call to
gm_provide_receive_buffer(), the client implicitly relinquishes a
receive token. With each buffer passed to
gm_provide_receive_buffer(), the client passes a corresponding
integer size indicating that the length of the receive buffer is
Before a client of a port can receive a message of a particular size and priority, the client software must provide GM with a receive token of matching size and priority. The receive token specifies the buffer in which to store the matching receive. When a message of matching size and priority is received, that message will be transferred into the receive buffer specified in the receive token. Note that multiple receive tokens of the same size and priority may be provided to the port.
After providing receive buffers with sizes matching the sizes of all
packets that potentially could be received, the client must poll for
receive events using a
gm_*receive*() function. (Most developers
who think polling is unacceptable in their application find that polling
is fine as long as they do it in a separate thread.) The
gm_*receive*() function will return a
The receipt of events of type
GM_HIGH_RECV_EVENT describe received packets of low and high
priority, respectively. All other events should be simply passed to
gm_unknown(). Such events are used internally by GM for sundry
purposes, and the client need not be concerned with the contents of
unrecognized receive events unless otherwise stated in this document.
To avoid deadlock of the port, the client software must ensure that the port is never without a receive token for any acceptable combination of size and priority for more than a bounded amount of time, that the port is informed which combinations of size and priority are not acceptable for receives, and that the client not send to any remote port that does not do likewise.
By convention, when a port runs out of low priority receive tokens for any combination of sizes, the client may defer replacing the receive tokens pending the completion of a bounded number of high priority sends, but must always replace exhausted types of high priority receive tokens without waiting for any sends to complete. Using this technique, reliable, deadlock-free, single-hop forwarding can be achieved.
Go to the first, previous, next, last section, table of contents.