Towards autonomic data management for staging-based coupled scientific workflows

Emerging scientific workflows running at extreme scale are composed of multiple applications that interact and exchange data at runtime. While staging-based approaches, e.g. in-situ/in-transit processing, are promising, dynamic behaviors (e.g. data volumes and distributions) in coupled applications and varying resource constraints at runtime make the efficient use of these techniques challenging. Addressing these challenges requires fundamental changes in the way that workflows are executed at runtime. Specifically, it is required to monitor the operating environment and running applications, and then adapt and tune the application behaviors and resource allocations at runtime while meeting the data management requirements and constraints. In this paper, we propose a policy-based autonomic data management (ADM) approach that can adaptively respond at runtime to dynamic data management requirements. We first formulate the schematic abstraction of this ADM approach including its conceptual model and system elements. Then, we explore the realization of ADM runtime and demonstrate how to achieve adaptations in a cross-layer manner with pre-defined autonomic policies. We also prototype our ADM approach and evaluate its performance on the Intrepid IBM-BlueGene and Titan Cray-XK7 systems using Chombo-based AMR applications and a visualization application. The experimental results demonstrate its effectiveness in meeting user defined objectives and accelerating overall scientific discovery. © 2020 Elsevier Inc.