XIDINTFL-VAE: XGBoost-based intrusion detection of imbalance network traffic via class-wise focal loss variational autoencoder

Intrusion Detection Systems (IDS) face significant challenges in detecting minority class attacks within imbalanced network traffic, where traditional methods often struggle to maintain high accuracy without sacrificing key performance metrics like precision and recall. This study introduces the XIDINTFL-VAE framework, which leverages Class-Wise Focal Loss (CWFL) and Variational AutoEncoder (VAE) integrated with XGBoost to effectively address this imbalance. Our approach is designed to enhance the detection of minority class intrusions while maintaining robust overall performance. Current techniques, such as SMOTE and its variants, often fail to balance precision and recall adequately in highly imbalanced datasets, resulting in either high false positives or missed detections. The proposed method directly addresses this gap by generating synthetic data tailored to the most challenging cases within the minority class, thereby improving the classifier's ability to detect these rare but critical instances. A comparative analysis of the proposed XIDINTFL-VAE method was conducted against various oversampling techniques, including SMOTE, Borderline-SMOTE, and Adaptive Synthetic Sampling (ADASYN), as well as traditional classifiers like Logistic Regression, K-Nearest Neighbors (KNN), Support Vector Machine (SVM), and Decision Tree. While existing techniques focus on balancing datasets or generating realistic data, the CWFL-VAE method takes a more targeted approach by generating synthetic data that specifically enhances the classifier's ability to handle challenging minority class instances. Experimental evaluations using the NSL-KDD and CSE-CIC-IDS2018 datasets demonstrate that the XIDINTFL-VAE model outperforms traditional methods, achieving a precision of 99.67% and an F1 score of 94.74%, with a slight trade-off in recall at 89.41%. These results underscore the model's capability to reduce false positives while maintaining high detection rates, which is crucial for real-world applications. The statistical significance of these improvements is confirmed by comparative analysis, establishing that our approach offers a meaningful advancement over existing methods.