Message passing with PVM(e)

PVMe is the IBM proprietary version of the widely used PVM message passing library from Oak Ridge National Laboratory. Its compatibility with the public domain PVM package generally lags one release behind. (For example, the current release of PVMe is compatible with PVM 3.2.6)

We will assume that the reader has a basic understanding of the concept of message passing communication of data on distributed memory parallel architectures. PVM(e) is described here primarily by example, but for the interested reader, extensive PVM documentation is available from the PVM authors at PVM on Netlib and from the online documentation available on danube.

Setting up the PVM environment

Before executing a PVM(e) message passing program, a ``Virtual Machine" (VM) must be initiated by the user. This is done by invoking what is known as the PVM(e) daemon, a process which sets up and maintains the information needed for PVM(e) processes to communicate with one another. In general, the user can ``customize" the VM by specifying which hosts it should include, the working directory for PVM processes on each host, the path to search for executables on each host, and so on. PVMe behaves slightly differently than PVM, since nodes are controlled through a Resource Manager. Rather than specifying particular nodes for the virtual machine, the user requests a certain number of nodes, and the Resource Manager reserves these nodes for that one particular user. This is to allow the user dedicated access to the High Performance Switch for the duration of their PVMe job.

One difference between PVM and PVMe on the SP2 is how information is supplied to the daemon. For PVM, a command line argument is used to pass the name of a ``hostfile". The PVM hostfile is a formatted text file which specifies this information with a simple syntax understood by PVM. An example is found in hostfile .

For PVMe, a command line flag to the daemon, ``-n < number > ", specifies the number of nodes to request from the Resource Manager. To override the default settings for the working directory and executable search path, environment variables can be used. See the PVMe description in /doc/pvme/ and the section Interactive tutorial examples for more information.

Checking the PVM environment

Both PVM and PVMe provide a utility for checking the status of the VM. This utility, called the PVM console, is started by entering the command ``pvm." A variety of query commands can then be entered at the console prompt to determine, for example, which hosts belong to the VM, which processes are running under PVM, etc. See the documentation PVM on Netlib for detailed information.

Getting started with PVM programming

With some exceptions (noted in section on differences between PVM and PVMe), PVM and PVMe are source code compatible. However, it is vital that your program is linked with the appropriate PVM or PVMe message passing libraries; these are NOT interchangeable. A code compiled and linked for PVM must be relinked to run under PVMe. See the section on Compiling message passing programs for sample makefiles with the proper library paths for linking PVM/PVMe programs. With this important difference noted, we will use PVM to refer to both PVM and PVMe throughout the remainder of this section,

In coding a message passing program in PVM, calls to communication routines from PVM are included in the same way one makes subroutine calls in Fortran, or function calls in C. A program which requires PVM communication must "enroll" itself in the PVM virtual machine before these calls are made. The call to enroll a program in PVM is:

call pvmfmytid(nodeid) or, in C: nodeid = pvm_mytid();

Sending a PVM message

Messages are prepared for sending by packing a buffer with the data items to be included in the message, with one ``pack" call for each data item. This allows the user to easily include data of varying types within the same message. A ``initialization" call is made just prior to packing to initialize the message buffer. Once packed, the buffer is sent to a specified destination with a "send" call. The initialization call serves primarily to set whether the data is to be converted to the portable XDR format prior to sending (first argument to pvminitsend is PVMDATADEFAULT, or 0), or sent in its raw form (first argument to pvminitsend is PVMDATARAW, or 1). A third possible value for the first argument to the initialization call is PVMDATAINPLACE, which eliminates the copying of the message into the buffer during packing and therefore improves efficiency. Packing in this context copies a pointer to the data and its size into the buffer (note: the data within each pack must occupy contiguous memory locations). When PVMDATAINPLACE is used, care must be taken not to modify the data before the message is actually received. For details about the trade-offs between these initialization schemes, the reader is referred to the PVM or PVMe reference guides. The examples provided in this primer use the PVMDATADEFAULT option, which give the highest degree of portability and protection from unintentional data overwrites.

Sample of Fortran message sending with PVM:

c  Initialize the buffer bufid:
      call pvmfinitsend(0, bufid)

c  Pack the real value rstart:
      call pvmfpack(REAL8, rstart, 1, 1, rc)

c  Pack the real 10-element vector data:
      call pvmfpack(REAL8, data, 10, 1, rc)

c  Pack the integer*4 value chunksize:
      call pvmfpack(INTEGER4, chunksize, 1, 1, rc)

c  Send the message to the node given in tids(1) with a 'tag' of 100:
      call pvmfsend(tids(1), 100, rc)

Sample of C message sending with PVM:

/* Initialize the buffer bufid (to use default XDR encoding): */
      bufid = pvm_initsend(PvmDataDefault)

/* Pack the real value rstart: */
      rc = pvm_pkdouble(&rstart, 1, 1)

/* Pack the real 10-element vector data: */
      rc = pvm_pkdouble(data, 10, 1)

/* Pack the integer*4 value chunksize: */
      rc = pvm_pkint(&chunksize, 1, 1)

/* Send the message to the node given in tids[0] with 'tag' 100:*/
      rc = pvm_send(tids[0], 100)

Receiving a PVM message

Messages can be received selectively based on a combination of the source (which node is sending the message) and tag (a qualifier specified by the user in the corresponding "send" call). A "wildcard" value (-1) can be specified for one or both of these receive call arguments to allow a message to be received regardless of the source or tag. When a message fitting the specified qualifications arrives at the node, it is received into a buffer. To retrieve the message contents, the buffer must be unpacked, in the same order in which it was packed at the sending end.

Sample of Fortran message receiving with PVM:

c  Receive a message from node hostid with tag 100 into a buffer:
      call pvmfrecv(hostid, 100, bufid)

c  Unpack the real value rstart:
      call pvmfunpack(REAL8, rstart, 1, 1, rc)

c  Unpack the real 10-element vector data:
      call pvmfunpack(REAL8, data, 10, 1, rc)

c  Unpack the integer*4 value chunksize:
      call pvmfunpack(INTEGER4, chunksize, 1, 1, rc)

Sample of C message receiving with PVM:

/* Receive a message from node hostid with tag 100 into a buffer: */
      pvm_recv(hostid, 100)

/* Unpack the real value rstart: */
      rc = pvm_upkdouble(&rstart, 1, 1)

/* Unpack the real 10-element vector data: */
      rc = pvm_upkdouble(data, 10, 1)

/* Unpack the integer*4 value chunksize: */
      rc = pvm_upkint(&chunksize, 1, 1)

Broadcasting a message in PVM

Work in progress...

Differences between PVM and PVMe

Work in progress...