TMaR: a two-stage MapReduce scheduler for heterogeneous environments

被引：8

作者：

Maleki, Neda ^{[1
]}

Faragardi, Hamid Reza ^{[2
]}

Rahmani, Amir Masoud ^{[3
]}

Conti, Mauro ^{[4
]}

Lofstead, Jay ^{[5
]}

机构：

[1] Islamic Azad Univ, Dept Comp Engn, Sci & Res Branch, Tehran, Iran

[2] KTH Royal Inst Technol, Dept Comp Sci & Commun, Stockholm, Sweden

[3] Khazar Univ, Dept Comp Sci, Baku, Azerbaijan

[4] Univ Padua, Dept Math, Padua, Italy

[5] Sandia Natl Labs, POB 5800, Albuquerque, NM 87185 USA

来源：

HUMAN-CENTRIC COMPUTING AND INFORMATION SCIENCES | 2020年 / 10卷 / 01期

关键词：

MapReduce; Hadoop; Heterogeneous systems; Scheduling; Performance; Shuffling; Power; Cloud computing; LOCALITY-AWARE; MAKESPAN; ALGORITHMS; SYSTEMS; TIME;

D O I：

10.1186/s13673-020-00247-5

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%.

引用

页数：26