The emergence and widespread adoption of the single program, multiple data (SPMD) programming model and the standardization of parallel communications libraries in the 1990s have increased the use of parallel computers and offered the carrot of the very highest performance for scientific computing. Another significant advance has been the availability of the message passing interface (MPI) standard. A program can now be both parallel and sufficiently independent of architectural details to be portable to a wide range of parallel environments, including shared-memory and distributed-memory multiprocessors, networks of workstations, and distributed cluster computers. In the case of the computational modeling of the flow properties of complex fluids, realistic simulations require many particles and hence large memory and long computation times. Parallel computing has allowed us to systematically explore regions of parameter space (e.g., different solid fractions, broader particle size and shape distributions) that would be prohibitive on single processor computers.

Two parallelizations have been created using MPI, a shared memory version and a distributed memory version. Data is replicated (the replicated data approach) in the shared memory version. In this approach every processor has a complete copy of all the arrays containing dynamical variables for every particle. The computation of forces is distributed over processors on the basis of cell indices. This is a very efficient way of implementing parallelism since the forces must be summed over processors only once per timestep, thus minimizing interprocessor communication costs. On shared-memory machines like an SGI Origin 2000, this approach is very attractive, since all processors can share the arrays containing dynamical variables.

On the other hand, the replicated data approach has turned out to be almost unusable on distributed memory systems including those with high speed interconnects like the IBM SP2/SP3 systems. Since the QDPD sequential code uses a link-cell algorithm which breaks simulation space into domains, it seems natural to map this geometrical, or domain, decomposition onto separate processors. Doing so is the essence of the parallel link-cell algorithm we have implemented. Our implementation has some fairly novel features arising from the DPD formalism (which forces some tricky bookkeeping to satisfy Newton's third law), the use of ellipsoids spread out across processors and the requirement of a sheared boundary condition (which causes particle movement of greater than one processor at times). This description is a good description of our spatial decomposition program, with the following additions to the description. First, following Plimpton, we distinguish between "owned" particles and "other" particles, those particles that are on neighboring processors and are part of the extended volume on any given processor. For "other" particles, only the information needed to calculate forces is communicated to neighboring processors. Second, the QDPD technique is being applied to suspensions, so there are two types of particles, "free" particles and particles belonging to ellipsoids. A novel feature of this work is that we explicitly do not keep all particles belonging to the same ellipsoid on the same processor. Since the largest ellipsoid that might be built can consist of as much as 50 percent of all particles, that would be difficult if not impossible to handle without serious load-balancing implications. What we do is assign each particle a unique particle number when it is read in. Each processor has the list of ellipsoid definitions consisting of lists of particles defined by these unique particle numbers. Each processor computes solid inclusion properties for each particle it "owns", and these properties are globally summed over all processors so that all processors have the same solid inclusion properties. Since there are only a small number of ellipsoids (relative to the number of particles), the amount of communication necessary for the global sums is small and the amount of extra memory is also relatively small. Hence it is an effective technique.

The replicated data approach has worked well for small to medium sized problems (tens-of-thousands of particles) on shared-memory SGIs. We have found speedups of as much as 17.5 on 24 processors of a 32 processor SGI Origin 2000. Using three such systems, we were able to get a year's worth of conventional computing done in a week.

For distributed memory systems, our spatial (domain) decomposition technique has proven to be effective. A parallel speedup of 24.19 was obtained for a benchmark calculation on 27 processors of an IBM SP3 cluster. Current results show a speedup of a factor of 22.5 on 27 200MHz Power3 processors on an IBM SP2/SP3 distributed memory system. THe same technique is also very effective in a shared-memory environment, where the speedups are a factor of 29 on 32 processors of an SGI Origin 3000 system and a factor of 50 on 64 processors.

Modeling the Flow of Suspensions in High Performance Concrete