Service-Aware Two-Level Partitioning for Machine Learning-Based Network Intrusion Detection With High Performance and High Scalability

A network intrusion detection system (NIDS) is an important technology for cyber security. Recently, machine learning based NIDSs are being actively researched as various machine learning techniques are proposed. However, existing NIDSs have limitation in terms of generality because they have been designed based on specific characteristics obtained from analyzing some partial datasets. Moreover, in reality, the NIDS datasets have a significantly imbalanced ratio between normal and abnormal data. It causes the minority class problem, which needs to be addressed for developing robust and reliable NIDSs working in various environments. This paper proposes a novel technique using service-aware dataset partitioning, which provides high scalability to handle huge and rapidly growing network data flexibly, and helps the classifier to improve the classification performance in terms of accuracy and speed. We evaluated our approach with the Kyoto2016 dataset, which is a well-known dataset for highly imbalanced data, using various classification algorithms and parameters for achieving the best performance and compared it with existing state-of-the-art approaches. Our experimental results indicated that our approach can classify network traffics rapidly and accurately with huge imbalanced datasets. We conclude that it can relieve serious existing issues of imbalanced datasets for modern machine learning based NIDS solutions.