Real Time Distributed Analysis of MPLS Network Logs for Anomaly Detection

Large scale IP networks contain thousands of network devices such as routers and switches. Massive amounts of logging data is generated by these devices. Analysing this data is both a challenge and an opportunity for finding network problems. Moreover, large IP networks contain devices from different vendors, so it is important to build a system which can work with network devices of different brands. In this study we describe a distributed architecture which can retrieve, store, and process massive amounts of network logging data in real time. Using this architecture we also build a basic anomaly detection system. The system statistically models cumulative counts of logs for different event types for all the devices in the network. The statistical approach lets the system to detect deviations from the normal behaviour without consulting expert knowledge. Our evaluations show that the system effectively handles massive amounts of data and detects anomalies.