Towards an Intelligent Framework for Scientific Computational Steering in Big Data Systems

Scientific applications of the next generation are undergoing a paradigm shift, transitioning from traditional experiment-centric methodologies to extreme-scale simulation-centric computations. These simulations, characterized by intricate numerical modeling with numerous adjustable parameters, generate vast datasets that necessitate meticulous processing and analysis against experimental or observational data for parameter calibration and model validation. However, manual parameter adjustment by domain experts in complex and distributed environments proves impractical. To address this challenge, we propose an online computational steering service facilitating real-time multi-user interaction. Towards this end, we design a versatile steering framework and conduct a theoretical performance evaluation of the steering service empowered by machine learning techniques. Furthermore, we present a case study involving the Weather Research and Forecast (WRF) model, comparing the performance of our steering solution with alternative heuristic methods and default settings to demonstrate its efficacy. The processing of big data generated by scientific simulations typically requires the use of big data systems as exemplified by Hadoop with Hadoop Distributed File System (HDFS) serving as a foundational technology layer. HDFS supports parallel computing in upper layers, offering fault tolerance and high throughput in data storage through block replication and cluster-wide distribution. However, the default block distribution strategy in HDFS overlooks the diverse capacities and data access patterns of nodes in heterogeneous Hadoop clusters, rendering it suboptimal for such environments. To address this issue, we formulate a class of block distribution problems in heterogeneous clusters, establishing its NP-completeness, and design an approximate algorithm, LPIR-BD, which leverages linear programming-based iterative rounding with a rigorous performance guarantee. Extensive experimental evaluations demonstrate the superior performance of LPIR-BD over several existing algorithms, corroborating our theoretical analyses and underscoring its efficacy in heterogeneous clusters.