Understanding the flow properties of complex fluids like
suspensions (e.g., colloids, ceramic slurries, and concrete) is
of importance and presents a significant theoretical challenge.
The computational modeling of such systems is also
challenging because of the difficulty of tracking boundaries between
different fluid/fluid and fluid/solid phases. Recently, a
new computational method called Dissipative Particle Dynamics (DPD)
has been introduced which has several advantages over traditional
computational dynamics methods while naturally accommodating
such boundary conditions. DPD resembles Molecular Dynamics (MD) in
that the particles move according to Newton's laws, but in DPD the
interparticle interactions are chosen to allow for much larger time
steps. This allows for the study of physical behaviour on time scales many
orders of magnitude greater than possible with MD. The original
DPD algorithm used an Euler algorithm for updating the positions
of the free particles (which represent "lumps" of fluids), and a
leap frog algorithm for updating the positions of the solid
inclusions. The NIST algorithm QDPD, for quarternionbased
dissipative particle dynamics, is a modification of the DPD
algorithm that uses a velocity Verlet algorithm to update the positions
of both the free particles and the solid inclusions. In addition, the
solid inclusion motion is determined from the quaternionbased scheme
of Omelayan (hence the Q in QDPD).
The computational modeling of the flow properties of complex fluids
like suspensions has been a great challenge because
of the difficulty of tracking boundaries between different fluid/fluid
and fluid/solid phases. QDPD naturally accomodates these boundary
conditions. QDPD in its present form it is being used to study the
steadyshear viscosity of a suspension of solid inclusions such as
ellipsoids in a Newtonian fluid, i.e., one in which time evolution is
governed by Newton's equations of motion.
The Quaternionbased Dissipative Particle Dynamics Method has been parallelized.

Why Parallelize Dissipative Particle Dynamics?
The emergence and widespread adoption of the single program,
multiple data (SPMD) programming model and the standardization of
parallel communications libraries in the 1990s have increased the
use of parallel computers and offered the carrot of the very highest
performance for scientific computing. Another significant advance has
been the availability of the message passing interface (MPI) standard.
A program can now be both parallel and sufficiently independent of
architectural details to be portable to a wide range of parallel
environments, including sharedmemory and distributedmemory
multiprocessors, networks of workstations, and distributed cluster
computers. In the case of the computational modeling of the flow
properties of complex fluids, realistic simulations require many
particles and hence large memory and long computation times.
Parallel computing has allowed us to systematically explore regions
of parameter space (e.g., different solid fractions, broader particle
size and shape distributions) that would be prohibitive on single
processor computers.


How is the Parallelization Realized?
Two parallelizations have been created using MPI, a shared memory
version and a distributed memory version. Data is replicated (the
replicated data approach) in the shared memory version. In this
approach every processor has a complete copy of all the arrays
containing dynamical variables for every particle. The computation of
forces is distributed over processors on the basis of cell indices.
This is a very efficient way of implementing parallelism since the
forces must be summed over processors only once per timestep, thus
minimizing interprocessor communication costs. On sharedmemory
machines like an SGI Origin 2000, this approach is very attractive,
since all processors can share the arrays containing dynamical
variables.
On the other hand, the replicated data approach has turned out
to be almost unusable on distributed memory systems including
those with high speed interconnects like the IBM SP2/SP3
systems. Since the QDPD sequential code uses a linkcell
algorithm which breaks simulation space into domains, it seems
natural to map this geometrical, or domain, decomposition onto
separate processors. Doing so is the essence of the parallel
linkcell algorithm we have implemented. Our implementation
has some fairly novel features arising from the DPD formalism
(which forces some tricky bookkeeping to satisfy Newton's third
law), the use of ellipsoids spread out across processors and the
requirement of a sheared boundary condition (which causes
particle movement of greater than one processor at times).
This description is a good description of our spatial
decomposition program, with the following additions to
the description. First, following Plimpton,
we distinguish between "owned" particles and "other" particles,
those particles that are on neighboring processors and are part
of the extended volume on any given processor. For "other"
particles, only the information needed to calculate forces is
communicated to neighboring processors. Second, the QDPD technique
is being applied to suspensions, so there are two types of particles,
"free" particles and particles belonging to ellipsoids. A novel
feature of this work is that we explicitly do not keep all
particles belonging to the same ellipsoid on the same processor.
Since the largest ellipsoid that might be built can consist of as much
as 50 percent of all particles, that would be difficult if not
impossible to handle without serious loadbalancing implications.
What we do is assign each particle a unique particle number when it
is read in. Each processor has the list of ellipsoid definitions
consisting of lists of particles defined by these unique particle
numbers. Each processor computes solid inclusion properties for each
particle it "owns", and these properties are globally summed over all
processors so that all processors have the same solid inclusion
properties. Since there are only a small number of ellipsoids
(relative to the number of particles), the amount of communication
necessary for the global sums is small and the amount of extra
memory is also relatively small. Hence it is an effective technique.


What is the Performance of the parallel Code?
The replicated data approach has worked well for small to medium
sized problems (tensofthousands of particles) on sharedmemory
SGIs. We have found speedups of as much as 17.5 on 24 processors
of a 32 processor SGI Origin 2000. Using three such systems,
we were able to get a year's worth of conventional computing
done in a week.
For distributed memory systems, our spatial (domain) decomposition
technique has proven to be effective. A parallel speedup of
24.19 was obtained for a benchmark calculation on 27 processors
of an IBM SP3 cluster. Current results show a speedup of a
factor of 22.5 on 27 200MHz Power3 processors on an IBM SP2/SP3
distributed memory system. THe same technique is also very
effective in a sharedmemory environment, where the speedups
are a factor of 29 on 32 processors of an SGI Origin 3000 system
and a factor of 50 on 64 processors.





Modeling the Flow of Suspensions in High Performance Concrete


Papers/Presentations

Edward Garboczi , Jeffrey Bullard, Nicos Martys and Judith Terrill,
The Virtual Cement and Concrete Testing Laboratory: Performance Prediction, Sustainability, and the CSHub
in NRMCA Concrete Sustainability Conference , Tempe, AZ,
April 13July 15, 2010.


Judith Terrill, W. L. George , Terrence J. Griffin , John Hagedorn, John T. Kelso, Marc Olano, Adele Peskin , S. Satterfield, James S. Sims, J. W. Bullard, Joy Dunkers, N. S. Martys , Agnes O'Gallagher and Gillian Haemer,
Extending Measurement Science to Interactive Visualization Environments
in Trends in Interactive Visualization: AStateoftheArt Survey,
Elena ZudilovaSeinstra, Tony Adriaansen and Robert Van Liere (Ed.)
,
Springer, U.K.,
2009.
Note: Pages: 207302 

Nicos Martys, Didier Lootens, W. L. George and Pascal Hebraud , Contact and stress anisotropies in startup flow of colloidal suspensions,
Physical Review E, 80,
2009.
ID: 031401.
Note: Comment: Phys. Rev. E80, 031401, 7 pages, (2009) 

James S. Sims and Nicos S. Martys, Simulation of Sheared Suspensions with a Parallel Implementation of QDPD,
Journal of Research of the National Institute of Standards and Technology, 109
(2)
,
pp. 267277.
Links:
pdf and postscript.


N. S. Martys , D. Lootens, W. L. George , S. Satterfield and P.Hebraud ,
SpatialTemporal Correlations in concentrated suspensions
in 15th International Congress on Rheology, Monterey CA.,
August 38, 2008.


N. S. Martys , D. Lootens, W. L. George , S. Satterfield and P.Hebraud ,
Stress chains formation under shear of concentrated suspension
in 15th International Congress on Rheology, Monterey CA.,
August 3, 2009August 8, 2008.


N. S. Martys , C. F. Ferraris , V. Gupta , J.H. Cheung , J. G. Hagedorn , A. P. Peskin and E. J. Garboczi ,
Computational Model Predictions of Suspension Rheology: Comparison to Experiment
in 12th International Conference on the Chemistry of Cement, Montreal, Canada,
July 813, 2007.


Nicos S. Martys and James S. Sims,
Modeling the Rheological Properties of Concrete
delivered at Virtual Cement and Concrete Testing Laboratory Meeting, Gaithersburg, MD,
June 8, 2000.


James S. Sims, William L. George, Steven G. Satterfield, Howard K. Hung, John G. Hagedorn, Peter M. Ketcham, Terence J. Griffin, Stanley A. Hagstrom, Julien C. Franiatte, Garnett W. Bryant, W. Jaskolski, Nicos S. Martys, Charles E. Bouldin, Vernon Simmons, Olivier P. Nicolas, James A. Warren, Barbara A. am Ende, John E. Koontz, B. James Filla, Vital G. Pourprix, Stefanie R. Copley, Robert B. Bohn, Adele P. Peskin, Yolanda M. Parker and Judith E. Devaney, Accelerating Scientific Discovery Through Computation
and Visualization II,
NIST Journal of Research, 107
(3)
,
MayJune, 2002,
pp. 223245.
Links:
postscript and pdf.


James S. Sims, John G. Hagedorn, Peter M. Ketcham, Steven G. Satterfield, Terence J. Griffin, William L. George, Howland A. Fowler, Barbara A. am Ende, Howard K. Hung, Robert B. Bohn, John E. Koontz, Nicos S. Martys, Charles E. Bouldin, James A. Warren, David L. Feder, Charles W. Clark, B. James Filla and Judith E. Devaney, Accelerating Scientific Discovery Through Computation
and Visualization ,
NIST Journal of Research, 105
(6)
,
NovemberDecember, 2000,
pp. 875894.
Links:
postscript and pdf.



