An Optimizing Algorithm for Deadline Constrained Scheduling of Scientific Workflows in IaaS Clouds Using Spot Instances

In recent years, due the advantages of on-demand resource provisioning and pay-as-you-go billing model, IaaS clouds are becoming more and more popular for performing large-scale scientific applications such as scientific workflows. To increase the utilization of the resources, many cloud vendors provide transient resources with charging price far below their normal counterparts, such as the spot instances provided by Amazon EC2. By using transient resources, the execution cost of scientific workflows can be greatly reduced. The side effect is that transient resources could be revoked at any time and hence may delay the execution of the scientific workflows. To overcome the problem, in this paper we propose an optimizing algorithm for deadline constrained scheduling of scientific workflows in IaaS clouds using spot instances. The proposed algorithm generates initial schedules based existing scheduling algorithms but adopt a partial schedule segmentation method to divide the initial schedules into diverse sections. For different sections, on-demand and spot instances are adaptively used according to the characteristics of the tasks in the sections. The experimental results of real-world workflow applications show that the proposed optimization algorithm can significantly reduce the monetary cost of the initial schedule solutions, while keeping their deadline constraints nearly all the time.