On the Implementation of Zigzag Codes for Distributed Storage System

被引：0

作者：

Lu, Lijia ^{[1
]}

Li, Hui ^{[1
]}

Chen, Jun ^{[1
]}

Zhu, Bing ^{[1
]}

Yin, Weijun ^{[2
]}

机构：

[1] Peking Univ, Shenzhen Grad Sch, Inst Big Data Technol, Shenzhen, Peoples R China

[2] Shenzhen Huadong Feitian Network Dev Co Ltd, Shenzhen, Peoples R China

来源：

PROCEEDINGS 2015 IEEE INTERNATIONAL CONFERENCE ON BIG DATA | 2015年

关键词：

Distributed Storage Systems; Zigzag Codes; Optimal Repair Property; Hadoop; SCHEME;

D O I：

暂无

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

Erasure codes such as Reed-Solomon (RS) codes are widely used to improve data reliability in distributed storage systems. Although erasure codes indeed greatly reduce the storage overhead compared to the replication schemes, it is still very costly in terms of network bandwidth when repairing a failed node. To address such problem, we employ the Zigzag codes, a MDS array code with optimal repair property, in the practical system. Specifically, we first build a general system on Hadoop to evaluate the encoding, decoding and repair performance of different codes, and then implement Zigzag codes on our system. The experimental results show that the Zigzag codes coincide with the theoretical findings and has certain advantages. Compared to current HDFS modules that use RS codes our Zigzag based HDFS implementation shows significant reduction of repair disk I/O and repair bandwidth with the same computation complexity.

引用

页码：1791 / 1796

页数：6

共 12 条

[1]

[Anonymous], 2003, P 19 ACM S OP SYST P, DOI [10.1145/1165389.945450, DOI 10.1145/1165389.945450]

[2] EVENODD - AN EFFICIENT SCHEME FOR TOLERATING DOUBLE-DISK FAILURES IN RAID ARCHITECTURES [J].

BLAUM, M ;

BRADY, J ;

BRUCK, J ;

MENON, J .

IEEE TRANSACTIONS ON COMPUTERS, 1995, 44 (02) :192-202

[3]

Cheng Huang, 2012, USENIX ANN TECHN C M, P15

[4]

Corbett P, 2004, USENIX ASSOCIATION PROCEEDINGS OF THE 3RD USENIX CONFERENCE ON FILE AND STORAGE TECHNOLOGIES, P1

[5] Network Coding for Distributed Storage Systems [J].

Dimakis, Alexandros G. ;

Godfrey, P. Brighten ;

Wu, Yunnan ;

Wainwright, Martin J. ;

Ramchandran, Kannan .

IEEE TRANSACTIONS ON INFORMATION THEORY, 2010, 56 (09) :4539-4551

[6] STAR: An efficient coding scheme for correcting triple storage node failures [J].

Huang, Cheng ;

Xu, Lihao .

IEEE TRANSACTIONS ON COMPUTERS, 2008, 57 (07) :889-901

[7]

Luby M., 1995, An xor-based erasure-resilient coding scheme

[8] Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction [J].

Rashmi, K. V. ;

Shah, Nihar B. ;

Kumar, P. Vijay .

IEEE TRANSACTIONS ON INFORMATION THEORY, 2011, 57 (08) :5227-5239

[9] XORing Elephants: Novel Erasure Codes for Big Data [J].

Sathiamoorthy, Maheswaran ;

Asteris, Megasthenis ;

Papailiopoulos, Dimitris ;

Dimakis, Alexandros G. ;

Vadali, Ramkumar ;

Chen, Scott ;

Borthakur, Dhruba .

PROCEEDINGS OF THE VLDB ENDOWMENT, 2013, 6 (05) :325-336

[10] Zigzag Codes: MDS Array Codes With Optimal Rebuilding [J].

Tamo, Itzhak ;

Wang, Zhiying ;

Bruck, Jehoshua .

IEEE TRANSACTIONS ON INFORMATION THEORY, 2013, 59 (03) :1597-1616

← 1 2 →