Optimization of Task Assignment Strategy for Map-Reduce

With the coming of this big data age, parallel processing is essential to processing a massive volume of data in a timely manner. Map-Reduce, which has been popularized, is a scalable and fault-tolerant data processing framework. It enables to process a massive volume of data in parallel way with many low-end computing nodes. As an important part of the framework, map task assignment has a significant impact on the performance of Map-Reduce. But in the allocation of the input files for map tasks, Map-Reduce framework does not take into account the distribution of the input data blocks in the file system and the load of the computing nodes themselves, which leading to increase the amount of network data transfer and system load when running map tasks. Especially when the framework uses the FIFO job scheduling strategy to deal with a large number of small jobs, the performance of the framework will be very low. In this paper, we design and implement a new task assignment strategy to increase the performance and efficiency of the Map-Reduce framework.