Data locality-aware and QoS-aware dynamic cloud workflow scheduling in Hadoop for heterogeneous environment

Hadoop has become a popular data-parallel computing framework for data-intensive scientific applications in recent years. Most scientific applications employ workflows to portray procedures and dependencies between jobs. However, the current default scheduling policy in Hadoop does not take data locality into account. The movement of data among virtual machines (VMs) produces latency in workflow scheduling. In addition, the heterogeneous and dynamics of cloud resources cannot satisfy the user's demand for quality of service (QoS) in static workflow scheduling. Hence, we propose a data locality-aware and QoS-aware dynamic cloud workflow scheduling algorithm (DQ-DCWS) based on dynamic programming. The algorithm balances data locality and delays by grouping nodes that hold tasks correlated with data blocks. We consider five QoS factors and normalise them as a path optimisation issue to realise maximum QoS. DQ-DCWS is implemented and validated by running Montage workflow on real Hadoop clusters which are deployed on Amazon EC2.