CARSS: Client-Aware Resource Sharing and Scheduling for Heterogeneous Applications

Modern hardware accelerators such as GP-GPUs and DSPs are commonly being used in real-time settings such as high-performance multimedia systems and autonomous vehicles. In fact, the throughput of a wide variety of computationally demanding tasks from 3D graphics and rendering to image processing and deep learning can benefit from such specialized hardware. Such heterogeneity can affect the performance of applications running simultaneously on the same accelerator. Prior studies on resource sharing and scheduling on hardware accelerators have not attempted to account for this context. In this work, we provide a portable tagging-based cooperative scheduler and resource monitor for use by heterogeneous applications sharing a single hardware accelerator in a soft real-time environment. We also offer practical insight into how various types of applications use the hardware accelerators differently. We substantiate the feasibility of our approach and evaluate the improvement of various scheduling policies over a proprietary scheduler in several case-studies with real-world applications on 2 NVIDIA platforms: a GeForce GTX 1070 GPU and an Xavier embedded platform(1). Although we focus on GPUs in this paper, our underlying observations and framework can also be used for sharing execution on other types of hardware accelerators.