LICOM3-CUDA: a GPU version of LASG/IAP climate system ocean model version 3 based on CUDA

The ocean general circulation model (OGCM) is an essential tool for researching oceanography and atmospheric science. The LASG/IAP climate system ocean model version 3 (LICOM3) is a parallel version of the OGCM. Our goal is to implement and optimize a GPU version of LICOM3 based on compute unified device architecture (CUDA) called LICOM3-CUDA. Considering the characteristics of LICOM3 and CUDA, we design and implement some pivotal optimization methods, including redesigning the numerical algorithms of complicated functions, decoupling data dependency, avoiding memory write conflicts, and optimizing communication. In this paper, we selected two experiments, including 1∘\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{\circ }$$\end{document} (small-scale) and 0.1∘\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{\circ }$$\end{document} (large-scale) resolutions to evaluate the performance of LICOM3-CUDA. Under the experimental environment of two Intel Xeon Gold 6148 CPUs and four NVIDIA Quadro GV100s, the LICOM3-CUDA (1∘\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{\circ }$$\end{document}) achieves a simulation speed of 114.3 simulation-year-per-day (SYPD). Compare with the performance of LICOM3, the LICOM3-CUDA can run much faster with 6.5 times, and the compute-intensive module achieves over 70×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times$$\end{document} speedup. In addition, the energy consumption for the simulation year is reduced by 41.3%. As for high-resolution and large-scale simulation, the number of GPUs increased from 96 to 1536 as well as the LICOM3-CUDA (0.1∘\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{\circ }$$\end{document}) time consumption decreased from 3261 to 720 seconds with approximately 4.5×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times$$\end{document} of speedup.