Design and Implementation of Intelligent Operation and Maintenance System for Big Data Platform

In recent years, with the development of big data technologies and applications, more and more enterprises realize that big data will play an important role in enterprises, which leads to a significant increase of big data platforms adopted by enterprises. Intelligent operation and maintenance could complete system state evaluation, abnormal diagnosis, alarm and prediction by means of data analysis, so as to reduce the complexity and workload of system operation and maintenance. In order to simplify the operation and maintenance of big data platform, intelligent operation and maintenance of big data platform has become a trend. In this context, the intelligent operation and maintenance system for big data platform is designed and implemented. The system realizes the collection, aggregation and storage, analysis with query and visualization of the runtime data of the big data platform. By providing a series of intelligent operation and maintenance functions, the complexity and workload of operation and maintenance personnel are reduced. In this thesis, the unified scoring model of big data platform is proposed; the HDFS capacity prediction method is proposed to manage the storage capacity of big data platform more actively; the task scheduling method based on platform free time analysis is proposed to optimize the platform efficiency.