Rough Logs: A Data Reduction Approach for Log Files

Modern scalable information systems produce a constant stream of log records to describe their activities and current state. This data is increasingly used for online anomaly analysis, so that dependability problems such as security incidents can be detected while the system is running. Due to the constant scaling of many such systems, the amount of processed log data is a significant aspect to be considered in the choice of any anomaly detection approach. We therefore present a new idea for log data reduction called ,rough logs'. It utilizes rough set theory for reducing the number of attributes being collected in log data for representing events in the system. We tested the approach in a large case study - the experiments showed that data reduction possibilities proposed by our approach remain valid even when the log information is modified due to anomalies happening in the system.