MapReduce Model Implementation on MPI Platform

With development of Multicore clusters the task-scheduling problem in heterogeneous cluster has become hot point of research. The method to solve this problem in Cloud computing is virtualization, which can make the heterogeneous nodes being isomorphic and then using MapReduce model for task scheduling in isomorphic nodes. But the approach has some shortcomings: virtualization itself will cause the loss of performance; and there are much more disk IOs in the MapReduce model, which can also cause performance degradation. Based on our earlier work which successfully adds fault-tolerance functions in MPI, this paper proposes a MPI based MapReduce approach which implements internodes communication with efficient MPI communication functions to achieve task scheduling on heterogeneous nodes directly by improved work pool and thread pool. By this way the load balancing can be achieved efficiency. The proposed MPI based MapReduce model can efficiently deal with a kind of data intensive as well as computation intensive problems.