A Fast Map-Reduce Algorithm for Burst Errors in Big Data Cloud Storage

In distributed storage for Big Data systems, there is a need for exact repair, high bandwidth codes. The challenge for exact repair in big-data storage is to simultaneously enable both very high bandwidth repair using Map-Reduce and simple coding schemes that also combine robust maximally distance separable (MDS) exact repair. MDS repair is for the rare, but exceptional outlier error patterns requiring optimum erasure code reconstruction. We construct the optimum fast bandwidth repair for big-data sources. Our system uses Map-Reduce, exact repair reconstruction. The algorithm combines MDS with a second fast decode algorithm in a cloud environment. We illustrate cloud experiments for optimum fast bandwidth reconstruction for 1-Exabyte Big Data in the cloud and demonstrate cloud results for Poisson error rate arrival models. Unlike prior methods, we jointly solve the problem of fast bandwidth repair for burst-memory error patterns and for code rates up to 2/3 in a real time error model framework for Big Data. Furthermore, simulations indicate this method outperforms prior fast bandwidth approaches for burst errors. We also illustrate Map-Reduce algorithm optimized for fast bandwidth repair in Big Data storage in clouds.