A Framework for Automated Parallel Execution of Scientific Multi-workflow Applications in the Cloud with Work Stealing

In this paper, we propose and evaluate an MPI/OpenMP framework to execute cloud applications composed of scientific linear multi-workflows with unknown task execution times and substantial I/O activity. In order to achieve load balancing, our framework incorporates a two-level work stealing strategy, with intra-node and inter-node stealing. The framework was evaluated in a cluster of 16 virtual machine (VM) instances (4 vCPUs), deployed on AWS Parallel Cluster. The results show that, for a real Bioinformatics application composed of 400 workflows, we are able to reduce the execution time from 1 h and 57 min (sequential) to 2min and 52 s (16 instances), achieving a speedup of 40.89x, with 64 threads.