Optimizing Task Assignment in Hadoop Using an Efficient Job Size-Based Scheduler

Hadoop is a distributed master-slave platform that comprises of two main components viz. Hadoop Distributed File System (HDFS) and MapReduce. HDFS provides distributed storage whereas MapReduce is useful for computational processing. A MapReduce cluster when receives multiple jobs simultaneously, the whole system performance might seriously deteriorate because of poor job response time. Thus, a real challenging issue in the MapReduce world is the efficient scheduling of jobs. Nevertheless, we see that traditional scheduling algorithms that work with Hadoop does not always assure significant average job-response times under distinct workloads. In order to address this problem, we put forward an efficient Hadoop scheduler that collects the information of workload patterns and distributes the jobs according to our hybrid scheduling technique. The experimental results exhibit that our scheduler enhances the average job-response time for MapReduce systems with different workload patterns.