Comparing Threshold Selection Methods for Network Anomaly Detection

The use of unsupervised machine learning models for anomaly detection is a common thing nowadays. While many research papers focus on improving and testing these models, there is a lack of those that deal with threshold selection, which is an important step in implementing a good anomaly detection system. In this paper, we investigate different supervised and unsupervised threshold selection methods found in the network anomaly detection literature. A total of five supervised and twenty unsupervised methods were found, all of which are described, categorized, and implemented in this paper. The unsupervised methods were further categorized according to the input data they expect, the type of output data they produce, and whether they are parametric or not, and divided into six groups according to the idea behind these methods: Statistics-based, Distribution-based, Clustering-based, Density-based, Graphical-based methods and Other. To test all the methods found, two different testing scenarios are created. The first one focuses on using data with anomalies and the second one uses only the normal data. Based on these two scenarios, tests were performed with real firewall log data containing three types of injected anomalies. The results are presented in the form of boxplots of the Matthews correlation coefficient for nine datasets. To draw a conclusion, both the method groups and the individual methods were compared in terms of evaluation metrics and execution times as well as in comparison to the methods already implemented in the PyThresh toolkit.