Research on Exception Data Cleaning Method Based on Clustering in Hadoop Platform

In the context of the explosive growth of data, the problem of data quality has become increasingly prominent, and it has become the main factor of data analysis and application. In order to get high-quality data, you need to clean the data. In this paper, the cleaning of the anomaly data is carried out. When the K-means clustering method is used for cleaning, the choice of the initial point is too dependent on the clustering result. At the same time,Large data sets of cleaning is easier to take much time becouse of the increasing numbers of operations. For solve these problems, this paper presents a method of anomaly data cleaning based on clustering under Hadoop platform. Firstly, this method improves the K-means algorithm by using the Canopy algorithm and the "minimum maximum principle" and the weighted Euclidean distance. Then it uses the MapReduce programming model to realize the parallelization of the method. The experimental results show that in the large data environment, using this method for abnormal data cleaning achieved good results whether in accuracy or in terms of speed up.