Exploring and cleaning big data with random sample data blocks

被引：15

作者：

Salloum, Salman ^{[1
,2
]}

Huang, Joshua Zhexue ^{[1
,2
]}

He, Yulin ^{[1
,2
]}

机构：

[1] Shenzhen Univ, Coll Comp Sci & Software Engn, Big Data Inst, Shenzhen 518060, Peoples R China

[2] Shenzhen Univ, Natl Engn Lab Big Data Syst Comp Technol, Shenzhen 518060, Peoples R China

来源：

JOURNAL OF BIG DATA | 2019年 / 6卷 / 01期

基金：

国家重点研发计划; 中国博士后科学基金; 中国国家自然科学基金;

关键词：

Big data; Exploratory data analysis; Statistical estimation; Data cleaning; Block-level sampling; Random sample partition; Distributed; Parallel and cluster computing;

D O I：

10.1186/s40537-019-0205-4

中图分类号：

TP301 [理论、方法];

学科分类号：

081202 ;

摘要：

Data scientists need scalable methods to explore and clean big data before applying advanced data analysis and mining algorithms. In this paper, we propose the RSP-Explore method to enable data scientists to iteratively explore big data on small computing clusters. We address three main tasks: statistical estimation, error detection, and data cleaning. The Random Sample Partition (RSP) distributed data model is used to represent the data as a set of ready-to-use random sample data blocks (called RSP blocks) of the entire data. Block-level samples of RSP blocks are selected to understand the data, identify potential types of value errors, and get samples of clean data. We provide a theoretical analysis on using RSP blocks for statistical estimation and demonstrate empirically the advantages of the RSP-Explore method. The experimental results of three real data sets show that the approximate results from RSP-Explore can rapidly converge toward the true values. Furthermore, cleaning a sample of RSP blocks is sufficient to estimate the statistical properties of the unknown clean data.

引用

页数：28

共 58 条

[31]

Krishnan Sanjay, 2015, IEEE Data Engineering Bulletin, V38, P59

[32] Visualization Viewpoints Sampling for Scalable Visual Analytics [J].