Resolved particle simulations using the Physalis method on many GPUs

We present a distributed memory many-GPU implementation of the Physalis method for resolving spherical particles in disperse multiphase flow simulations. The current work extends a previous single-GPU computational procedure by implementing a distributed memory Poisson solver and distributed finite-size particle methods using MPI. We document the changes required to move to a distributed memory model for both the fluid and solid phases. We benchmark the code with up to one million resolved particles in a domain size of 19203 on 216 GPUs at the Maryland Advanced Research Computing Center and present strong and weak scaling results. Finally, by taking advantage of the realization that the solution procedure for the pressure Poisson equation can be implemented using a symmetric matrix, we are able to replace the biconjugate gradient stabilized algorithm used in previous work with the conjugate gradient algorithm. (C) 2019 Published by Elsevier B.V.