Dynamic Erasure Coding Policy Allocation (DECPA) in Hadoop 3.0

被引：8

作者：

Chiniah, Aatish ^{[1
]}

Mungur, Avinash ^{[2
]}

机构：

[1] Univ Mauritius, Dept Digital Technol, Reduit, Mauritius

[2] Univ Mauritius, Dept ICT, Reduit, Mauritius

来源：

2019 6TH IEEE INTERNATIONAL CONFERENCE ON CYBER SECURITY AND CLOUD COMPUTING (IEEE CSCLOUD 2019) / 2019 5TH IEEE INTERNATIONAL CONFERENCE ON EDGE COMPUTING AND SCALABLE CLOUD (IEEE EDGECOM 2019) | 2019年

关键词：

Erasure Code; Replication; Cloud Storage; Network Attached Storage; Hadoop; 3.0;

D O I：

10.1109/CSCloud/EdgeCom.2019.00015

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

Erasure Code (EC) is being tipped to be the next best alternative to Replication for providing redundancy for Cloud Storage. Already major players like Microsoft and Facebook are having initial implementations using Erasure Code. Hadoop 0.20 was the first version that supported Erasure Code (aka HDFS-RAID), but EC was not included in later versions, only to resurface in Hadoop 3.0 (HDFS-EC). Hadoop 3.0.0 supports three default Erasure Code polices which are RS(3,2), RS(6,3) and RS(10,4). To have greater flexibility, in this work we opt for the implementation of new Erasure Code policies [RS(4,3), RS(5,3), RS(7,3), RS(7,4), RS(8,4), RS(9,4)] and the development of a Dynamic Erasure Coding Policy Allocation, based on minimum overhead produced, in order to maximize storage capacity. Three types of dynamic allocation have been proposed and implemented. A performance evaluation of the new polices was conducted in order to find the optional one for a NAS/SAN architecture and the effectiveness of the three implemented Dynamic Allocation of EC policy is provided.

引用

页码：29 / 33

页数：5

共 16 条

[1]

Apache, 2018, AP HAD 3 0 0 ALPH SN

[2] Target Tracking with Limited Sensing Range in Autonomous Mobile Sensor Networks [J].

Bai, Jing ;

Cheng, Peng ;

Chen, Jiming ;

Guenard, Adrien ;

Song, Yeqiong .

2012 IEEE 8TH INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING IN SENSOR SYSTEMS (DCOSS), 2012, :329-334

[3]

Chen, USENIX ATC 2017

[4]

Chiniah A., 2019, ADV INFORM COMMUNICA, V887

[5] Erasure-Coded Network Backup System (ECNBS) [J].

Chiniah, Aatish ;

Dhora, Jellina Aishwarta Devi ;

Sandooram, Chaahana Juhinee .

INFORMATION, COMMUNICATION AND COMPUTING TECHNOLOGY, 2017, 750 :35-43

[6]

Esmaili KS, 2013, IEEE INT CONF BIG DA

[7]

Hadoop. apache. org, 2018, AP HAD 3 0 0 HDFS ER

[8]

Hadoop. apache. org, 2018, AP HAD 2 9 1 RACK AW

[9] A Dynamic Data Placement Strategy for Hadoop in Heterogeneous Environments [J].

Lee, Chia-Wei ;

Hsieh, Kuang-Yu ;

Hsieh, Sun-Yuan ;

Hsiao, Hung-Chang .

BIG DATA RESEARCH, 2014, 1 :14-22

[10]

Li X., 2016, IEEE RAD C RAD C, P1, DOI [10.1109/ICCRE.2016.7476145, DOI 10.1109/ICCRE.2016.7476145]

← 1 2 →