I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

被引：11

作者：

Bez, Jean Luca ^{[1
]}

Tang, Houjun ^{[1
]}

Xie, Bing ^{[2
]}

Williams-Young, David ^{[1
]}

Latham, Rob ^{[3
]}

Ross, Rob ^{[3
]}

Oral, Sarp ^{[2
]}

Byna, Suren ^{[1
]}

机构：

[1] Lawrence Berkeley Natl Lab, Berkeley, CA 94720 USA

[2] Oak Ridge Natl Lab, Oak Ridge, TN USA

[3] Argonne Natl Lab, Lemont, IL USA

来源：

PROCEEDINGS OF IEEE/ACM SIXTH INTERNATIONAL PARALLEL DATA SYSTEMS WORKSHOP (PDSW 2021) | 2021年

关键词：

D O I：

10.1109/PDSW54622.2021.00008

中图分类号：

TP3 [计算技术、计算机技术];

学科分类号：

0812 ;

摘要：

Using parallel file systems efficiently is a tricky problem due to inter-dependencies among multiple layers of I/O software, including high-level I/O libraries (HDF5, netCDF, etc.), MPI-IO, POSIX, and file systems (GPFS, Lustre, etc.). Profiling tools such as Darshan collect traces to help understand the I/O performance behavior. However, there are significant gaps in analyzing the collected traces and then applying tuning options offered by various layers of I/O software. Seeking to connect the dots between I/O bottleneck detection and tuning, we propose DXT Explorer, an interactive log analysis tool. In this paper, we present a case study using our interactive log analysis tool to identify and apply various I/O optimizations. We report an evaluation of performance improvement achieved for four I/O kernels extracted from science applications.

引用

页码：15 / 22

页数：8

共 25 条

[11] CAPES: Unsupervised Storage Performance Tuning Using Neural Network-Based Deep Reinforcement Learning
Li, Yan
Chang, Kenneth
Bel, Oceane
Miller, Ethan L.
Long, Darrell D. E.
[J]. SC'17: PROCEEDINGS OF THE INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS, 2017,
[12] Lofstead J.F., 2008, P 6 INT WORKSH CHALL, P15, DOI 10.1145/1383529.1383533
[13] Lofstead J, 2011, HPDC 11: PROCEEDINGS OF THE 20TH INTERNATIONAL SYMPOSIUM ON HIGH PERFORMANCE DISTRIBUTED COMPUTING, P49
[14] The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science
Marek, A.
Blum, V.
Johanni, R.
Havu, V.
Lang, B.
Auckenthaler, T.
Heinecke, A.
Bungartz, H-J
Lederer, H.
[J]. JOURNAL OF PHYSICS-CONDENSED MATTER, 2014, 26 (21)
[15] A User-Friendly Approach for Tuning Parallel File Operations
McLay, Robert
James, Doug
Liu, Si
Cazes, John
Barth, William
[J]. SC14: INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS, 2014, : 229 - 236
[16] P.T. Inc, 2015, COLL DAT SCI
[17] Revisiting I/O Behavior in Large-Scale Storage Systems: The Expected and the Unexpected
Patel, Tirthak
Byna, Surendra
Lockwood, Glenn K.
Tiwari, Devesh
[J]. PROCEEDINGS OF SC19: THE INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS, 2019,
[18] Foundations of JSON']JSON Schema
Pezoa, Felipe
Reutter, Juan L.
Suarez, Fernando
Ugarte, Martin
Vrgoc, Domagoj
[J]. PROCEEDINGS OF THE 25TH INTERNATIONAL CONFERENCE ON WORLD WIDE WEB (WWW'16), 2016, : 263 - 273
[19] Prost Jean-Pierre., 2001, SUPERCOMPUTING 01, P17
[20] Data sieving and collective I/O in ROMIO
Thakur, R
Gropp, W
Lusk, E
[J]. FRONTIERS '99 - THE SEVENTH SYMPOSIUM ON THE FRONTIERS OF MASSIVELY PARALLEL COMPUTATION, PROCEEDINGS, 1999, : 182 - 189

← 1 2 3 →