Task Completion Time Prediction Scaled by Machine Learning Model Uncertainty

We discuss "GPU Time Sharing," which involves sharing GPU servers locally owned in the private cloud to increase their usage and to reduce the costs of cloud services. In the GPU Time Sharing, users can reserve and use a CPU server for a certain period of time to complete a task, such as training the AI model or inference on a video. In the reservation, task completion time must be predicted conservatively to ensure that the task is completed in the reserved time, but at the same time, the reserved time should be as short as possible for efficient resource sharing. In this paper, we propose a task completion time prediction method called "Uncertainty Scaled Gradient Boosting Decision Tree" (USGBDT), which first predicts the completion time of the Deep Learning (DL) tasks using the Gradient Boosting Decision Tree, and then scales the predicted time based on the expected uncertainty of the machine learning models. Applying the proposed method to the CPU Time Sharing for video analysis tasks, we have confirmed that all tasks are completed in the predicted completion time and the GPU usage time over the reserved time is improved from 53.4% to 67.0%.