An Efficiency-Aware Scheduling for Data-Intensive Computations on MapReduce Clusters

Scheduling plays a key role in Map Reduce systems. In this paper, we explore the efficiency of an Map Reduce cluster running lots of independent and continuously arriving Map Reduce jobs. Data locality and load balancing are two important factors to improve computation efficiency in Map Reduce systems for data-intensive computations. Traditional cluster scheduling technologies are not well suitable for Map Reduce environment, there are some in-used schedulers for the popular open-source Hadoop Map Reduce implementation, however, they can not well optimize both factors. Our main objective is to minimize total fiowtime of all jobs, given it's a strong NP-hard problem, we adopt some effective heuristics to seek satisfied solution. In this paper, we formalize the scheduling problem as job selection problem, a load balance aware job selection algorithm is proposed, in task level we design a strict data locality tasks scheduling algorithm for map tasks on map machines and a load balance aware scheduling algorithm for reduce tasks on reduce machines. Comprehensive experiments have been conducted to compare our scheduling strategy with well-known Hadoop scheduling strategies. The experimental results validate the efficiency of our proposed sdheduling strategy.