Characterizing Multi-Instance GPU for Machine Learning Workloads

As machine learning (ML) becomes more and more popular, datacenter operators use hardware accelerators such as GPUs to tackle the high computation demand of ML workloads. However, recent studies show that user-submitted jobs often underutilize the GPU streaming multiprocessor (SM) cores, resulting in hardware resource wastage. Motivated by this observation, GPU vendors have released software and hardware support for GPU resource sharing, for example, the NVIDIA Multi-Instance GPU (MIG) technique on A100 Tensor Core GPUs. In this work, we use several state-of-the-art deep learning (DL) models from various application areas to characterize the performance and energy consumption of the A100 GPU MIG mode operation. Our characterization reveals valuable insights into operating a MIG-enabled GPU datacenter.