Fault Tolerance Performance Evaluation of Large-Scale Distributed Storage Systems HDFS and Ceph Case Study

Large-scale distributed systems are a collection of loosely coupled computers interconnected by a communication network. They are now an integral part of everyday life with the development of large web applications, social networks, peer-to-peer systems, wireless sensor networks and many more. At such a scale, hardware components by themselves are prone to failure. Therefore, one key challenge in designing distributed storage systems is how to tolerate faults. To this end, fault tolerance mechanisms such as replication have been widely used to provide high availability for decades. More recently, many systems start supporting erasure coding for fault tolerance, which is expected to achieve high reliability at a lower storage cost compared to replication. However, the reduced storage overhead comes at the cost of more complicated recovery which hurts performance. In this paper, we study the fault tolerance mechanisms of two representative distributed file systems: HDFS and Ceph. In addition to the traditional replication, both HDFS and Ceph support erasure coding in their latest version. We evaluate the replication and erasure coding implementations in both systems using standard benchmarks and fault injection, and quantitatively measure the performance and storage overhead. Our results demonstrate the trade-offs between replication and erasure coding techniques, and serve as a foundation for building optimal storage systems with high availability as well as high performance.