Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers

被引:36
作者
Cheng, Long [1 ,2 ]
Wang, Ying [2 ]
Liu, Qingzhi [3 ]
Epema, Dick H. J. [4 ]
Liu, Cheng [2 ]
Mao, Ying [5 ]
Murphy, John [6 ]
机构
[1] North China Elect Power Univ, Sch Control & Comp Engn, Beijing 102206, Peoples R China
[2] Chinese Acad Sci, Inst Comp Technol, State Key Lab Comp Architecture, Beijing 100190, Peoples R China
[3] Wageningen Univ & Res, Informat Technol Grp, NL-6708 PB Wageningen, Netherlands
[4] Delft Univ Technol, Distributed Syst Grp, NL-2628 CD Delft, Netherlands
[5] Fordham Univ, Dept Comp & Informat Sci, New York, NY 10458 USA
[6] Univ Coll Dublin, Sch Comp Sci, Dublin D04 V1W8, Ireland
基金
中国国家自然科学基金; 欧盟地平线“2020”;
关键词
Distributed databases; Bandwidth; Scheduling; Data centers; Optimization; Processor scheduling; Big Data; Data locality; coflow scheduling; distributed operators; data centers; big data; SDN; metaheuristic; SOFTWARE-DEFINED NETWORKING; OPTIMIZATION; METAHEURISTICS; FUTURE; QOS; SDN;
D O I
10.1109/TPDS.2021.3053241
中图分类号
TP301 [理论、方法];
学科分类号
081202 ;
摘要
Large data centers are currently the mainstream infrastructures for big data processing. As one of the most fundamental tasks in these environments, the efficient execution of distributed data operators (e.g., join and aggregation) are still challenging current data systems, and one of the key performance issues is network communication time. State-of-the-art methods trying to improve that problem focus on either application-layer data locality optimization to reduce network traffic or on network-layer data flow optimization to increase bandwidth utilization. However, the techniques in the two layers are totally independent from each other, and performance gains from a joint optimization perspective have not yet been explored. In this article, we propose a novel approach called NEAL (NEtwork-Aware Locality scheduling) to bridge this gap, and consequently to further reduce communication time for distributed big data operators. We present the detailed design and implementation of NEAL, and our experimental results demonstrate that NEAL always performs better than current approaches for different workloads and network bandwidth configurations.
引用
收藏
页码:1494 / 1510
页数:17
相关论文
共 60 条
[1]  
Ahmad Faraz, 2014, Proceedings of USENIX ATC '14: 2014 USENIX Annual Technical Conference. ATC '14, P1
[2]   A scalable, commodity data center network architecture [J].
Al-Fares, Mohammad ;
Loukissas, Alexander ;
Vahdat, Amin .
ACM SIGCOMM COMPUTER COMMUNICATION REVIEW, 2008, 38 (04) :63-74
[3]  
[Anonymous], 2014, ACM CIKM INT C INF K
[4]  
[Anonymous], 2014, Proceedings of the 2014 ACM Conference on SIGCOMM
[5]   Task scheduling techniques in cloud computing: A literature survey [J].
Arunarani, A. R. ;
Manjula, D. ;
Sugumaran, Vijayan .
FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2019, 91 :407-415
[6]   Adaptive and Big Data Scale Parallel Execution in Oracle [J].
Bellamkonda, Srikanth ;
Li, Hua-Gang ;
Jagtap, Unmesh ;
Zhu, Yali ;
Liang, Vince ;
Cruanes, Thierry .
PROCEEDINGS OF THE VLDB ENDOWMENT, 2013, 6 (11) :1102-1113
[7]  
Blanas S., 2010, ACM SIGMOD INT C MAN, P975, DOI [DOI 10.1145/1807167.1807273, 10]
[8]   Metaheuristics in combinatorial optimization: Overview and conceptual comparison [J].
Blum, C ;
Roli, A .
ACM COMPUTING SURVEYS, 2003, 35 (03) :268-308
[9]   A survey on optimization metaheuristics [J].
Boussaid, Ilhern ;
Lepagnot, Julien ;
Siarry, Patrick .
INFORMATION SCIENCES, 2013, 237 :82-117
[10]   Advanced Join Strategies for Large-Scale Distributed Computation [J].
Bruno, Nicolas ;
Kwon, YongChul ;
Wu, Ming-Chuan .
PROCEEDINGS OF THE VLDB ENDOWMENT, 2014, 7 (13) :1484-1495