Framework for Genetic Algorithms Using Pilot Jobs in Adaptive Grid Workflows

The performance of Grid applications may be very unstable, especially when using workflows for job distribution. This is mainly due to the Grid overheads, like scheduling and queuing, introduced before the job is executed on a worker node. Optimization problems using Genetic Algorithms (GAs) can be easily and efficiently implemented on Grids using Grid workflows. Due to the file dependencies introduced in the Grid workflows for GAs, mainly for genetic material interchange, these overheads are cumulative and thus very noticeable. This problem is also very evident when the jobs are short compared to the Grid overheads, i.e. the job spends more time waiting in a queue to be executed than the execution itself. In this paper we introduce a framework that enables users to easily utilize the Grid infrastructure for their optimization using GAs. It allows a user to preallocate certain number of pilot jobs, and also to dynamically manage their number for optimal availability of resources during the optimization process. In this way, once an application starts to execute the workloads, it will have at least one available pilot for execution of pooled tasks. This introduces better utilization of the Grid resources, as well boost the confidence in the infrastructure from users point of view.