Robust Task Scheduling Strategy for Big Data Clusters

Task scheduling problem has been an active area in big data framework. Arriving tasks should be assigned to suitable nodes in a big data cluster for load balancing. The system can become more efficient when the workload is well distributed among different nodes. However, the existing researches consider little about the uncertainties in the resource and service provision. Since effective processor speed is changing over time, the computational ability of a node is not consistent. The load balancing may not be well achieved due to the computing uncertainty. In this paper, we propose a robust task scheduling algorithm for big data clusters. The computational uncertainty is modeled as perturbation on the processing speed, and our task scheduling approach is designed to deal with the potential computing uncertainties. The simulation results demonstrate that our scheduling strategy can reject perturbation and provide the stable computing service.