An Optimization Algorithm for Heterogeneous Hadoop Clusters Based on Dynamic Load Balancing

被引：0
|
作者：
Yan, Wei [1 ]
Li, ChunLin [1 ]
Du, ShuMeng [1 ]
Mao, Xijun [1 ]
机构：
[1] Wuhan Univ Technol, Software Engn, 1186 Heping Blvd, Wuhan, Hubei, Peoples R China
来源：
2016 17TH INTERNATIONAL CONFERENCE ON PARALLEL AND DISTRIBUTED COMPUTING, APPLICATIONS AND TECHNOLOGIES (PDCAT) | 2016年
关键词：
Hadoop; heterogeneous cluster; data allocation; load balancing;
D O I：
10.1109/PDCAT.2016.60
中图分类号：
TP3 [计算技术、计算机技术];
学科分类号：
0812 ;
摘要：
Hadoop is a popular cloud computing software, and its major component MapReduce can efficiently complete parallel computing in homogeneous environment. But in practical application heterogeneous cluster is a common phenomenon. In this case, it's prone to unbalance load. To solve this problem, a model of heterogeneous Hadoop cluster based on dynamic load balancing is proposed in this paper. This model starts from MapReduce and tracks node information in real time by using its monitoring module. A maximum node hit rate priority algorithm (MNHRPA) is designed and implemented in the paper, and it can achieve load balancing by dynamic adjustment of data allocation based on nodes' computing power and load. The experimental results show that the algorithm can effectively reduce tasks' completion time and achieve load balancing of the cluster compared with Hadoop's default algorithm.
引用
收藏
页码：250 / 255
页数：6
相关论文
共 50 条
  • [1] A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters
    Liu, Yang
    Jing, Weizhe
    Liu, Youbo
    Lv, Lin
    Qi, Man
    Xiang, Yang
    CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE, 2017, 29 (03):
  • [2] Dynamic Load Balancing Algorithm for Heterogeneous Clusters
    do Nascimento, Tiago Marques
    dos Santos, Rodrigo Weber
    Lobosco, Marcelo
    PARALLEL PROCESSING AND APPLIED MATHEMATICS (PPAM 2017), PT II, 2018, 10778 : 166 - 175
  • [3] Design of dynamic load balancing algorithm for heterogeneous clusters based on energy consumption
    Ye, Lunqiang
    ENERGY REPORTS, 2021, 7 : 684 - 695
  • [4] Dynamic load balancing on heterogeneous clusters for parallel ant colony optimization
    Antonio Llanes
    José M. Cecilia
    Antonia Sánchez
    José M. García
    Martyn Amos
    Manuel Ujaldón
    Cluster Computing, 2016, 19 : 1 - 11
  • [5] Dynamic load balancing on heterogeneous clusters for parallel ant colony optimization
    Llanes, Antonio
    Cecilia, Jose M.
    Sanchez, Antonia
    Garcia, Jose M.
    Amos, Martyn
    Ujaldon, Manuel
    CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS, 2016, 19 (01): : 1 - 11
  • [6] A load index and load balancing algorithm for heterogeneous clusters
    Jose Luis Bosque
    Pablo Toharia
    Oscar D. Robles
    Luis Pastor
    The Journal of Supercomputing, 2013, 65 : 1104 - 1113
  • [7] A load index and load balancing algorithm for heterogeneous clusters
    Luis Bosque, Jose
    Toharia, Pablo
    Robles, Oscar D.
    Pastor, Luis
    JOURNAL OF SUPERCOMPUTING, 2013, 65 (03): : 1104 - 1113
  • [8] A Dynamic Load Balancing Algorithm in Heterogeneous Network
    Ding, Zhixiong
    Wang, Xingjun
    Yang, Wenming
    2016 7TH INTERNATIONAL CONFERENCE ON INTELLIGENT SYSTEMS, MODELLING AND SIMULATION (ISMS), 2016, : 337 - 342
  • [9] Service Dependency Based Dynamic Load Balancing Algorithm for Container Clusters
    Zhang, Jianxin
    Ren, Rui
    Huang, Chengxi
    Fei, Xiang
    Qun, Wu
    Cai, Hongming
    2018 IEEE 15TH INTERNATIONAL CONFERENCE ON E-BUSINESS ENGINEERING (ICEBE 2018), 2018, : 70 - 77
  • [10] The research on dynamic load balancing algorithm for heterogeneous systems
    Zhu, Peiyou
    Gao, Ji
    Chen, Rong
    Zhang, Fan
    Guo, Hang
    WCICA 2006: SIXTH WORLD CONGRESS ON INTELLIGENT CONTROL AND AUTOMATION, VOLS 1-12, CONFERENCE PROCEEDINGS, 2006, : 4420 - +
12345