LogMaster: Mining Event Correlations in Logs of Large-scale Cluster Systems

被引：36

作者：

Fu, Xiaoyu ^{[1
]}

Ren, Rui ^{[1
]}

Zhan, Jianfeng ^{[1
]}

Zhou, Wei ^{[1
]}

Jia, Zhen ^{[1
]}

Lu, Gang ^{[1
]}

机构：

[1] Chinese Acad Sci, Inst Comp Technol, State Key Lab Comp Architecture, Beijing, Peoples R China

来源：

2012 31ST INTERNATIONAL SYMPOSIUM ON RELIABLE DISTRIBUTED SYSTEMS (SRDS 2012) | 2012年

关键词：

reliability; correlation mining; failure prediction; large-scale systems;

D O I：

10.1109/SRDS.2012.40

中图分类号：

TP3 [计算技术、计算机技术];

学科分类号：

0812 ;

摘要：

This paper presents a set of innovative algorithms and a system, named LogMaster, for mining correlations of events that have multiple attributions, i.e., node ID, application ID, event type, and event severity, in logs of large-scale cloud and HPC systems. Different from traditional transactional data, e.g., supermarket purchases, system logs have their unique characteristics, and hence we propose several innovative approaches to mining their correlations. We parse logs into an n-ary sequence where each event is identified by an informative nine-tuple. We propose a set of enhanced apriori-like algorithms for improving sequence mining efficiency; we propose an innovative abstraction-event correlation graphs (ECGs) to represent event correlations, and present an ECGs-based algorithm for fast predicting events. The experimental results on three logs of production cloud and HPC systems, varying from 433490 entries to 4747963 entries, show that our method can predict failures with a high precision and an acceptable recall rates.

引用

页码：71 / 80

页数：10