RAS: A Task Scheduling Algorithm Based on Resource Attribute Selection in a Task Scheduling Framework

With the advent of big data and cloud computing era, scheduling and executing large-scale computing tasks effectively and allocating resources to tasks reasonably are becoming a quite challenging problem. And there is theoretical significance to research on efficient scheduling algorithm to improve resource utilization and task execution efficiency. We present a scheduling algorithm based on resource attribute selection (RAS) by sending a set of test tasks to an execution node to determine its resource attributes before a task is scheduled; and then selecting the optimal node to execute a task according to its resource requirements and the fitness between the resource node and the task, which also uses history task data if exists. We (1) give a formal definition of the resource attributes and (2) compute the fitness of the resource nodes and (3) store the information of node selection for next round. We integrate our algorithm into the Gearman scheduling framework, and through comparison with three other scheduling frameworks, we find out there is significant improvement in resource selection and resource utilization using RAS. The throughput of the RAS (with work-stealing, WS) is at least 30% higher than the other frameworks and the resource utilization of RAS (WS) reaches 0.94. The algorithm can make a good model for practical large scale application scheduling.