Data provenance management and application in Clinical Data Spaces

In the era of big data, management of the increasing volume of clinical data has become a research hotspot. To manage the massive data in clinical data centers, and in order to clarify the data source, data iteration and data flow, we propose a data provenance management method based on a three-tier structure, which divides the management of data lineages into a data layer, a semantic layer, and a presentation layer. The method implemented in this paper collects all the commands executed by the operating system, the logs generated by each process, and the physical locations and relevant attributes of all data objects such as data tables, data files, dictionary files, web pages, etc. The semantic layer processes relevant information and generates a data lineage log based on the PROV standard proposed by W3C, and presented it at the presentation layer. Our method records the source, process, and storage location of each process, which enabled process replay. At the same time, our method semantically annotates each data object, which can better describe the source of data items, processing process, etc. Our method enables the replay of the analysis process and the retrospective analysis of the output process. And can be widely used in the management of massive clinical data.