A Smart Speculative Execution Strategy based on Node Classification for Heterogeneous Hadoop Systems

被引：0

作者：

Liu, Qi ^{[1
]}

Cai, Weidong ^{[1
]}

Shen, Jian ^{[1
]}

Fu, Zhangjie ^{[1
]}

Linge, Nigel ^{[2
]}

机构：

[1] Nanjing Univ Informat Sci & Technol, 219 Ningliu Rd, Nanjing 210044, Jiangsu, Peoples R China

[2] Univ Salford, Salford M5 4WT, Greater Manches, England

来源：

2016 18TH INTERNATIONAL CONFERENCE ON ADVANCED COMMUNICATIONS TECHNOLOGY (ICACT) - INFORMATION AND COMMUNICATIONS FOR SAFE AND SECURE LIFE | 2016年

关键词：

MapReduce; speculative execution; time prediction; Node classification;

D O I：

暂无

中图分类号：

TN [电子技术、通信技术];

学科分类号：

0809 ;

摘要：

MapReduce (MR) has been widely used to process distributed large data sets. Meanwhile, speculative execution is known as an approach for dealing with same problems by backing up those tasks running on a low performance machine to a higher one. In this paper, we have modified some pitfalls and taken heterogeneous environment into consideration. We also have implemented it in Hadoop-2.6 based on node classification, this strategy is called Speculation-NC and optimized Hadoop is called Hadoop-NC. Experiment results show that our method can correctly backup a task, improve the performance of MRV2 and decrease the execution time and resource consumption compared with traditional strategy.

引用

页码：223 / 227

页数：5

共 8 条

[1]

Ahmad F, 2012, ASPLOS XVII: SEVENTEENTH INTERNATIONAL CONFERENCE ON ARCHITECTURAL SUPPORT FOR PROGRAMMING LANGUAGES AND OPERATING SYSTEMS, P61

[2] A View of Cloud Computing [J].

Armbrust, Michael ;

Fox, Armando ;

Griffith, Rean ;

Joseph, Anthony D. ;

Katz, Randy ;

Konwinski, Andy ;

Lee, Gunho ;

Patterson, David ;

Rabkin, Ariel ;

Stoica, Ion ;

Zaharia, Matei .

COMMUNICATIONS OF THE ACM, 2010, 53 (04) :50-58

[3] Improving MapReduce Performance Using Smart Speculative Execution Strategy [J].

Chen, Qi ;

Liu, Cheng ;

Xiao, Zhen .

IEEE TRANSACTIONS ON COMPUTERS, 2014, 63 (04) :954-967

[4]

Dean J, 2004, USENIX ASSOCIATION PROCEEDINGS OF THE SIXTH SYMPOSIUM ON OPERATING SYSTEMS DESIGN AND IMPLEMENTATION (OSDE '04), P137

[5] Achieving Efficient Cloud Search Services: Multi-Keyword Ranked Search over Encrypted Cloud Data Supporting Parallel Computing [J].

Fu, Zhangjie ;

Sun, Xingming ;

Liu, Qi ;

Zhou, Lu ;

Shu, Jiangang .

IEICE TRANSACTIONS ON COMMUNICATIONS, 2015, E98B (01) :190-200

[6]

Jing Tai Piao, 2012, Web Technologies and Applications. Proceedings of the 14th Asia-Pacific Web Conference, APWeb 2012, P570, DOI 10.1007/978-3-642-29253-8_51

[7]

Matsunaga Andrea., 2010, 2010 10 IEEEACM INT, P495, DOI [DOI 10.1109/CCGRID.2010.98, 10.1109/CCGRID.2010.98]

[8]

Zaharia M., 2008, OSDI, V8

← 1 →