Batch job utilities: Extoll


Introduction to Extoll

Extoll provides a convenient "point-and-click" style interface for generating a LoadLeveler command file matched to a user's specific parallel program. (Note that Extoll superceeds the utilities X-llcreate/llcreate, providing more robust and streamlined services.)

Note: when using Extoll, your DISPLAY environment variable must be set to your working console. To accomplish this, type "setenv DISPLAY console.internet.address:0.0" at the unix prompt on danube, and "xhost +danube.nist.gov" at the unix prompt on your console.

When Extoll is invoked, the user is prompted for information about the communication library used (PVMe, MPL, or Oak Ridge PVM), the style of their program (SPMD or MPMD), arguments or input required, batch class and number of processors, and whether a custom setup or cleanup script is required. Finally, when the "Create *.cmd File" button is pressed, a command file is generated based on the input information.

Use of Extoll

To use Extoll, simply invoke it in the directory where your executables are stored (this is often the same as your source directory, or ~/pvm3/bin/RS6K). check the appropriate boxes and fill in requested information in the entry panel. The script is relatively generous in interpreting your input; for example, the program is insensitive to the case of most responses. In fact, most input can be handled with a click of a button, eliminating the need to worry about typos in the input.

Of course, with the use of this utility, you maintain the flexibility of editing the generated command file, which could be used, for example, to correct any mistyped program names, etc. You may wish to experiment with various combinations of responses to the queries, studying resulting command file, to help your understanding of how information is relayed to the generic script. If you run into difficulty because your particular program doesn't fit any of the standard scenarios handled by Extoll and the appropriate generic script file, please send mail to kremington@nist.gov and we will try to update the scripts to accommodate your needs.

A sample view of the Extoll interface:

(click to enlarge)

A short (8-page) user's guide is available, see Extoll: Easy X-interface to LoadLeveler for Parallel Jobs on the NIST IBM SP2, or the file /usr/local/apps/EXTOLL/doc/extoll-doc.ps on danube.

Generic Batch Scripts for Message Passing Programs

The Extoll utilitly relies on the use of generic batch scripts, one for each of the available message passing libraries. These scripts are intended to be system supported and maintained, so that all users benefit from locally developed improvements in batch script handling. Actual support of these scripts may vary. Please contact the author at kremington@nist.gov for current information.

Sample command file/script

#!/bin/ksh
# -----------------------------------------------------------------
# LoadLeveler Command Script:  main.cmd
# Automatically generated at:  Fri Aug 28 14:20:21 EDT 1998
# EXTOLL V. 2.0
# -----------------------------------------------------------------
#
# @ job_name = main
# @ initialdir = /danube/nist/karin/EXTOLL/bin
# @ output = $(initialdir)/$(Cluster).out
# @ error = $(initialdir)/$(Cluster).err
# @ notification = Error
# @ notify_user = karin@danube.nist.gov
# @ class = Small
# @ job_type = parallel
# @ min_processors = 4
# @ max_processors = 16
# @ environment = CLUSTER=$(Cluster); LL_JOB=TRUE
# @ restart = no
# @ checkpoint = no
# @ requirements = (Adapter == "hps_user")
# @ queue
#
# -----------------------------------------------------------------
#    Set up and start an SP2MPI PVM job which uses the SP2
#    switch in User Space mode (high performance).
#
#    NOTES:  > Executables must be compiled according to the
#              PVM documentation in README.mp.
#            > Processes are limited to one per node, and each
#              pvm_spawn generates an "extra" process.
#
#    (Please see PVM documentation in README.mp for more info.)
# -----------------------------------------------------------------
#    This stub is to set PVM_ROOT and PVM_ARCH in your .cshrc file.
#    See the EXTOLL documentation for proper .cshrc setup.
#
cp $EXTOLL_ROOT/bin/.pvm-switch.stub $HOME/.llpvmstub
# -----------------------------------------------------------------
#
#    Create a PVM hostfile with the Executable path set to /tmp/karin
#             ( using switch addresses hssw*.nist.gov)
#
cat /tmp/$LOADL_STEP_OWNER.$LOADL_STEP_ID.hostfile |
    sed -e 's/grand/hssw/' >> $CLUSTER.hostfile
# -----------------------------------------------------------------
export PVM_ROOT=$HOME/pvm3
export PVM_ARCH=SP2MPI
export PVM_PROCESSOR_LIST=$LOADL_PROCESSOR_LIST
export STARTHOST=`hostname | sed -e 's/grand/hssw/'`
export LOADL_PROCESSOR_LIST=`echo $LOADL_PROCESSOR_LIST |
                             sed -e 's/grand/hssw/g'`
export MP_PROCS=4
export MP_RMPOOL=1
export ARGUMENTS=""
export STANDARD_IN=""
export FILES_IN="my.data"
# -----------------------------------------------------------------
#    Copy executables and 'cd' to working directory:
# -----------------------------------------------------------------
for host in $PVM_PROCESSOR_LIST
do
  rsh $host "ksh -c '[ -d /tmp/karin ] || mkdir /tmp/karin'"
  rsh $host "cd /danube/nist/karin/EXTOLL/bin; cp -f main worker /tmp/karin"
done
cd /tmp/karin
# -----------------------------------------------------------------
#    Now, invoke the daemon and run the [setup] program [cleanup]:
# -----------------------------------------------------------------
$PVM_ROOT/lib/$PVM_ARCH/pvmd3 -n$STARTHOST $CLUSTER.hostfile &
sleep 15
/danube/nist/karin/EXTOLL/bin/user.setup
main $ARGUMENTS < $STANDARD_IN
/danube/nist/karin/EXTOLL/bin/user.cleanup
# -----------------------------------------------------------------
#    Delete executables and other files from working directory:
# -----------------------------------------------------------------
cd /danube/nist/karin/EXTOLL/bin
for host in $PVM_PROCESSOR_LIST
do
  rsh $host 'rm -r /tmp/karin"'
done
# -----------------------------------------------------------------
#    Finally, halt the daemon and copy back the PVM log file:
# -----------------------------------------------------------------
$PVM_ROOT/lib/$PVM_ARCH/pvm <