A Fast Map-Reduce Algorithm for Burst Errors in Big Data Cloud Storage

被引:0
作者
Qin, Xue [1 ]
Kelley, Brian [1 ]
Saedy, Mahdy [2 ]
机构
[1] Univ Texas San Antonio, Dept ECE, San Antonio, TX 78249 USA
[2] AT&T Res Labs, AT&T Serv Design & Dev, Middletown, NJ 07748 USA
来源
2015 10TH SYSTEM OF SYSTEMS ENGINEERING CONFERENCE (SOSE) | 2015年
关键词
Exabyte Big Data; fast reconstruction; exact repair; erasure codes; Map-Reduce; cloud systems; maximally distance separable;
D O I
暂无
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
In distributed storage for Big Data systems, there is a need for exact repair, high bandwidth codes. The challenge for exact repair in big-data storage is to simultaneously enable both very high bandwidth repair using Map-Reduce and simple coding schemes that also combine robust maximally distance separable (MDS) exact repair. MDS repair is for the rare, but exceptional outlier error patterns requiring optimum erasure code reconstruction. We construct the optimum fast bandwidth repair for big-data sources. Our system uses Map-Reduce, exact repair reconstruction. The algorithm combines MDS with a second fast decode algorithm in a cloud environment. We illustrate cloud experiments for optimum fast bandwidth reconstruction for 1-Exabyte Big Data in the cloud and demonstrate cloud results for Poisson error rate arrival models. Unlike prior methods, we jointly solve the problem of fast bandwidth repair for burst-memory error patterns and for code rates up to 2/3 in a real time error model framework for Big Data. Furthermore, simulations indicate this method outperforms prior fast bandwidth approaches for burst errors. We also illustrate Map-Reduce algorithm optimized for fast bandwidth repair in Big Data storage in clouds.
引用
收藏
页码:398 / 403
页数:6
相关论文
共 11 条
[1]  
Dean J, 2004, USENIX ASSOCIATION PROCEEDINGS OF THE SIXTH SYMPOSIUM ON OPERATING SYSTEMS DESIGN AND IMPLEMENTATION (OSDE '04), P137
[2]  
Hadoop A., 2009, PRO HADOOP
[3]  
Itoh Y., 1989, 1989 IEEE International Solid-State Circuits Conference. Digest of Technical Papers. 36th ISSCC. First Edition (Cat. No.CH2684-9), P134, DOI 10.1109/ISSCC.1989.48209
[4]  
Kelley Brian, 2014, Provisional Patent, Patent No. [62/090,868, 62090868]
[5]  
PAPAILIOPOULOS DS, 2012, 31 ANN IEEE INT C CO, P2801
[6]  
Qin Xiuchang, 2014, THESIS
[7]  
Shvachko K., 2010, 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), P1
[8]  
Wang Ailan, 2010, Proceedings 2010 3rd International Conference on Intelligent Networks and Intelligent Systems (ICINIS 2010), P381, DOI 10.1109/ICINIS.2010.118
[9]  
Wang G., 2008, ISI 2008 IEEE INT C, P245
[10]  
Weatherspoon H., 2002, 1 INT WORKSH PEER TO