Performance-related Internal Clustering Validation Index for Clustering-based Anomaly Detection

One possible way to improve unsupervised anomaly detection is to use per-cluster models, particularly when the given data includes various cluster-level features. In realizing clustering-based anomaly detection, one natural question is how to determine the number of clusters that will likely lead to the optimal performance. In this paper, we propose a method that reflects the performance of anomaly detection in determining the number of clusters. We first derive an internal clustering validation index using the normality scores of trained per-cluster models for unlabeled training data for cases with different numbers of clusters. Then, we determine the number of clusters by selecting the case whose clustering validation index is the highest, which means that per-cluster models extract useful features for anomaly detection. Through experiments, we show that our proposed clustering validation index is highly correlated with anomaly detection accuracy (i.e., the average Pearson correlation coefficient is 0.965).