HDMF: Hierarchical Data Modeling Framework for Modem Science Data Standards

被引：0

作者：

Tritt, Andrew J. ^{[1
]}

Rubel, Oliver ^{[1
]}

Dichter, Benjamin ^{[2
]}

Ly, Ryan ^{[1
]}

Kang, Donghe ^{[3
]}

Chang, Edward E. ^{[5
,6
]}

Frank, Loren M. ^{[4
]}

Bouchard, Kristofer ^{[2
]}

机构：

[1] Lawrence Berkeley Natl Lab, Computat Res Div, Berkeley, CA 94720 USA

[2] Lawrence Berkeley Natl Lab, Biol Syst & Engn, Berkeley, CA USA

[3] Ohio State Univ, Comp Sci & Engn, Columbus, OH 43210 USA

[4] Univ Calif San Francisco, Howard Hughes Med Inst, Kavli Inst Fundamental Neurosci, Dept Physiol, San Francisco, CA USA

[5] Univ Calif San Francisco, Dept Neurol Surg, San Francisco, CA USA

[6] Univ Calif San Francisco, Ctr Integrat Neurosci, San Francisco, CA 94143 USA

来源：

2019 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA) | 2019年

基金：

美国国家卫生研究院;

关键词：

data standards; data modeling; data formats; HDF5; neurophysiology;

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

A ubiquitous problem in aggregating data across different experimental and observational data sources is a lack of software infrastructure that enables flexible and extensible standardization of data and metadata. To address this challenge, we developed HDMF, a hierarchical data modeling framework for modern science data standards. With HDMF, we separate the process of data standardization into three main components: (1) data modeling and specification, (2) data I/O and storage, and (3) data interaction and data APIs. To enable standards to support the complex requirements and varying use cases throughout the data life cycle, HDMF provides object mapping infrastructure to insulate and integrate these various components. This approach supports the flexible development of data standards and extensions, optimized storage backends, and data APIs, while allowing the other components of the data standards ecosystem to remain stable. To meet the demands of modern, large-scale science data, HDMF provides advanced data I/O functionality for iterative data write, lazy data load, and parallel I/O. It also supports optimization of data storage via support for chunking, compression, linking, and modular data storage. We demonstrate the application of HDMF in practice to design NWB 2.0 [13], a modern data standard for collaborative science across the neurophysiology community.

引用

页码：165 / 179

页数：15

共 21 条

[11] NETCDF - AN INTERFACE FOR SCIENTIFIC-DATA ACCESS [J].

REW, R ;

DAVIS, G .

IEEE COMPUTER GRAPHICS AND APPLICATIONS, 1990, 10 (04) :76-82

[12]

Rubel O., 2019, bioRxiv

[13]

Rubel O., 2019, NWB N FORMAT SPECIFI

[14] OpenMSI: A High-Performance Web-Based Platform for Mass Spectrometry Imaging [J].

Ruebel, Oliver ;

Greiner, Annette ;

Cholia, Shreyas ;

Louie, Katherine ;

Bethel, E. Wes ;

Northen, Trent R. ;

Bowen, Benjamin P. .

ANALYTICAL CHEMISTRY, 2013, 85 (21) :10354-10361

[15] Methods for Specifying Scientific Data Standards and Modeling Relationships with Applications to Neuroscience [J].

Rulbel, Oliver ;

Dougherty, Max ;

Prabhat ;

Denes, Peter ;

Conant, David ;

Chang, Edward F. ;

Bouchard, Kristofer .

FRONTIERS IN NEUROINFORMATICS, 2016, 10

[16]

Sarawagi S., 1994, Proceedings. The 10th International Conference Data Engineering (Cat. No.94CH3383-7), P328, DOI 10.1109/ICDE.1994.283048

[17]

Shasharina S., 2009, IADIS INT C COMP GRA, P49

[18]

Stoewer A., 2019, NIX

[19] Usage Pattern-Driven Dynamic Data Layout Reorganization [J].

Tang, Houjun ;

Byna, Suren ;

Harenberg, Steve ;

Zou, Xiaocheng ;

Zhang, Wenzhao ;

Wu, Kesheng ;

Dong, Bin ;

Rubel, Oliver ;

Bouchard, Kristofer ;

Klasky, Scott ;

Samatova, Nagiza F. .

2016 16TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND GRID COMPUTING (CCGRID), 2016, :356-365

[20]

The HDF Group, 1997, HIER DAT FORM VERS 5

← 1 2 3 →