RAISE: Efficient GPU Resource Management via Hybrid Scheduling

As the de facto high-throughput accelerators, graphics processing units (GPUs) are now used in a wide spectrum of fields, including artificial intelligence, high performance computing and finance. While with excessive computing and memory resources, GPUs are facing significant challenges to reach high utilization by a monolithic task. Multiple tasks are thus concurrently running to share the GPUs, but they may adversely affect each other, causing performance degradation. As a result, it is extremely critical to manage resources in a reasonable way to strike a balance between utilization and performance. Targeting the issue, this paper proposes an effective resource management design via hybrid task scheduling. Our design continuously tracks the GPU executions and collects the usage statistics, which are then used to direct the task selection and dispatch, including the type, starting time and kernel dimensions. A prototype is developed on off-the-shelf GPUs by moderately refactoring the CUDA source codes. Experimental results show that the design can achieve up to 1.96x performance improvement (1.51x on average), meanwhile effectively boosting resource utilization.