Multi-GPU solution to the lattice Boltzmann method: An application in multiscale digital rock simulation for shale formation

Characterization of rock properties is vital in producing oil and gas from shale reservoirs in an economically viable fashion. The nano-pore structure and ultralow permeability in shale reservoirs present challenges to the traditional experimental characterization methods. Digital rock physics for the estimation of rock properties, especially for shale reservoirs, has become a powerful tool that greatly complements to lab experiments by combining advance imaging techniques with numerical simulations. The lattice Boltzmann method (LBM) is a well-applied numerical method to simulate the fluid flow in pore structures at multiple length scales. Usually, the LBM simulation is resource intense because of its computation complexity and is facing great numerical challenges in extremely large-cale computation. In this paper, we propose a multi-GPU parallel implementation of 3D LBM on a hybrid high-performance computing cluster to perform large-scale simulations in reconstructed digital rocks. The program provides multiscale solution, pore scale and representative elementary volume (REV) scale based on the resolution of digital rock images. Optimization strategies are applied on partitioning simulation domain, improving data communication efficiency and maximizing CUDA occupancy. When running on a cluster of 32 GPUs, the proposed parallel implementation achieves a speedup of 1074x comparing to the in-house sequential program.