Online Adaptive Anomaly Detection for Augmented Network Flows

Traditional network anomaly detection involves developing models that rely on packet inspection. However, increasing network speeds and use of encrypted protocols make per-packet inspection unsuited for today's networks. One method of overcoming this obstacle is aggregating packet header information and performing flow-based analysis where data flow patterns are examined rather than deep packet inspection. Many existing approaches are special purpose limited to detecting specific behavior. Also, the data reduction inherent in identifying anomalous flows hinders alert correlation. In this article, we propose and develop a dynamic anomaly detection approach for augmented network flows. We sketch network state during flow creation, enabling general-purpose threat detection. We describe an efficient flow augmentation approach based on the count-min sketch that provides per-flow-, per-node-, and per-network-level statistics parallel to flow record generation. We design and develop a support vector machine-based adaptive anomaly detection and correlation mechanism, which is capable of aggregating alerts without a priori alert classification and evolving models online. We further develop a lightweight evolving alert aggregation method and combine it with a confidence forwarding mechanism identifying a small percentage predictions for additional processing. We show effectiveness of our methods on both enterprise and backbone traces. Experimental results demonstrate its ability to maintain high accuracy without the need for offline training.