Rule-based Data Quality Intelligent Monitoring System

The rapid development of Internet information technology brought great convenience to people's lives and work, but it also brought the problem of information overload[1]. The data center is the middle layer that mediates the contradiction between the foreground and the background[2]. It provides a reusable, standardized, and agile multifunctional platform through the modeling of background data and the aggregation of data services to support the foreground for the demand[3] of quickly change according to market changes. Through the data center, we have opened up the data islands of each business system. However, when each business system uses data, it found that the data quality is poor, unstable, not timely, and inconsistent. Traditional cluster data quality monitoring has the characteristics of inconsistent data quality requirements for various services, difficult development, low processing efficiency, difficult maintenance, and insufficient intelligence. Based on this situation, we design a data quality monitoring system based on rules and artificial intelligence. General users perform simple configuration on the interface to complete real-time online monitoring of data, and real-time statistics on the number of data quality problems and a detailed list for users to trace the cause of the problem and identify abnormal scenarios in advance. The comparison between experiments and mainstream products proves that the system has greatly improved in terms of user use and flexibility.