A speculative approach to spatial-temporal efficiency with multi-objective optimization in a heterogeneous cloud environment

被引:187
作者
Liu, Qi [1 ]
Cai, Weidong [1 ]
Shen, Jian [2 ]
Fu, Zhangjie [3 ]
Liu, Xiaodong [4 ]
Linge, Nigel [5 ]
机构
[1] Nanjing Univ Informat Sci & Technol, Sch Comp & Software, Nanjing, Jiangsu, Peoples R China
[2] Nanjing Univ Informat Sci & Technol, Jiangsu Engn Ctr Network Monitoring, Nanjing, Jiangsu, Peoples R China
[3] Nanjing Univ Informat Sci & Technol, Jiangsu Collaborat Innovat Ctr Atmospher Environm, Nanjing, Jiangsu, Peoples R China
[4] Edinburgh Napier Univ, Sch Comp, 10 Colinton Rd, Edinburgh EH10 5DT, Midlothian, Scotland
[5] Univ Salford, Sch Comp Sci & Engn, Salford, Lancs, England
关键词
MapReduce; cloud storage; load balancing; multi-objective optimization; prediction model; EXTREME LEARNING-MACHINE; DATA PLACEMENT;
D O I
10.1002/sec.1582
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
A heterogeneous cloud system, for example, a Hadoop 2.6.0 platform, provides distributed but cohesive services with rich features on large-scale management, reliability, and error tolerance. As big data processing is concerned, newly built cloud clusters meet the challenges of performance optimization focusing on faster task execution and more efficient usage of computing resources. Presently proposed approaches concentrate on temporal improvement, that is, shortening MapReduce time, but seldom focus on storage occupation; however, unbalanced cloud storage strategies could exhaust those nodes with heavy MapReduce cycles and further challenge the security and stability of the entire cluster. In this paper, an adaptive method is presented aiming at spatial-temporal efficiency in a heterogeneous cloud environment. A prediction model based on an optimized Kernel-based Extreme Learning Machine algorithm is proposed for faster forecast of job execution duration and space occupation, which consequently facilitates the process of task scheduling through a multi-objective algorithm called time and space optimized NSGA-II (TS-NSGA-II). Experiment results have shown that compared with the original load-balancing scheme, our approach can save approximate 47-55 s averagely on each task execution. Simultaneously, 1.254 parts per thousand of differences on hard disk occupation were made among all scheduled reducers, which achieves 26.6% improvement over the original scheme. Copyright (C) 2016 John Wiley & Sons, Ltd.
引用
收藏
页码:4002 / 4012
页数:11
相关论文
共 30 条
  • [1] Hadoop-GIS: A High Performance Spatial Data Warehousing System over MapReduce
    Aji, Ablimit
    Wang, Fusheng
    Vo, Hoang
    Lee, Rubao
    Liu, Qiaoling
    Zhang, Xiaodong
    Saltz, Joel
    [J]. PROCEEDINGS OF THE VLDB ENDOWMENT, 2013, 6 (11): : 1009 - 1020
  • [2] Algebraic Optimization for Processing Graph Pattern Queries in the Cloud
    Anyanwu, Kemafor
    Kim, HyeongSik
    Ravindra, Padmashree
    [J]. IEEE INTERNET COMPUTING, 2013, 17 (02) : 52 - 61
  • [3] A View of Cloud Computing
    Armbrust, Michael
    Fox, Armando
    Griffith, Rean
    Joseph, Anthony D.
    Katz, Randy
    Konwinski, Andy
    Lee, Gunho
    Patterson, David
    Rabkin, Ariel
    Stoica, Ion
    Zaharia, Matei
    [J]. COMMUNICATIONS OF THE ACM, 2010, 53 (04) : 50 - 58
  • [4] Efficient co-processor utilization in database query processing
    Bress, Sebastian
    Beier, Felix
    Rauhe, Hannes
    Sattler, Kai-Uwe
    Schallehn, Eike
    Saake, Gunter
    [J]. INFORMATION SYSTEMS, 2013, 38 (08) : 1084 - 1096
  • [5] Robustness Against the Decision-Maker's Attitude to Risk in Problems With Conflicting Objectives
    Bui, Lam T.
    Abbass, Hussein A.
    Barlow, Michael
    Bender, Axel
    [J]. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, 2012, 16 (01) : 1 - 19
  • [6] Improving MapReduce Performance Using Smart Speculative Execution Strategy
    Chen, Qi
    Liu, Cheng
    Xiao, Zhen
    [J]. IEEE TRANSACTIONS ON COMPUTERS, 2014, 63 (04) : 954 - 967
  • [7] CoHadoop: Flexible Data Placement and Its Exploitation in Hadoop
    Eltabakh, Mohamed Y.
    Tian, Yuanyuan
    Ozcan, Fatma
    Gemulla, Rainer
    Krettek, Aljoscha
    McPherson, John
    [J]. PROCEEDINGS OF THE VLDB ENDOWMENT, 2011, 4 (09): : 575 - 585
  • [8] Esteves R, 2012, ACM SIGARCH COMPUTER, P61
  • [9] Esteves R. M., 2011, Proceedings 2011 25th IEEE International Conference on Advanced Information Networking and Applications Workshops (WAINA 2011), P514, DOI 10.1109/WAINA.2011.136
  • [10] Improving MapReduce Performance by Balancing Skewed Loads
    Fan Yuanquan
    Wu Weiguo
    Xu Yunlong
    Chen Heng
    [J]. CHINA COMMUNICATIONS, 2014, 11 (08) : 85 - 108