MIGER: Integrating Multi-Instance GPU and Multi-Process Service for Deep Learning Clusters

Modern NVIDIA GPUs, known for their powerful computational abilities, have been widely adopted by data centers. These GPUs often use space-sharing techniques, such as Multi-Process Service (MPS) and Multi-Instance GPU (MIG), to run multiple workloads on a GPU concurrently. However, our findings reveal that there are issues such as performance interference and inflexible resource size for these techniques when they are used individually. We present MIGER, a system that leverages both MPS and MIG techniques for online and offline jobs on modern GPUs. MIGER employs a hierarchical scheduling architecture to determine the sizes of MIG partitions, how to co-locate online and offline jobs, and the resource shares of MPS for each job to increase the throughput of offline jobs while guaranteeing the QoS requirements of online jobs. Through extensive real-cluster experiments, MIGER demonstrates a significant improvement in job completion time by 36% and 46.6% compared to the state-of-the-art MIG-based and MPS-based solutions, respectively.