AutoGuard: A Dual Intelligence Proactive Anomaly Detection at Application-Layer in 5G Networks

Application-layer protocols are widely adopted for signaling in telecommunication networks such as the 5G networks. However, they can be subject to application-layer attacks that are hardly detected by existing traditional network-based security tools that often do not support telecommunication-specific applications. To address this issue, we propose in this work AutoGuard, a proactive anomaly detection solution that employs application-layer Performance Measurement (PM) counters to train two different Deep Learning (DL) techniques, namely, Long Short Term Memory (LSTM) networks and AutoEncoders (AEs). We leverage recent advancements in Machine Learning (ML) that show the advantages brought by combining multiple ML models to build a dual-intelligence approach allowing the proactive detection of application layer anomalies. Our proposed dual-intelligence solution promotes signaling workload forecasting and anomaly prediction as a proactive security control in 5G networks. As a proof of concept, we implement our approach for the proactive detection of Diameter-related signaling attacks on the Home Subscriber Server (HSS) core network function. To evaluate our solution, we conduct a set of experiments using data collected from a real 5G testbed. Our results show the effectiveness of our dual intelligence approach on proactively detecting signaling anomalies with a precision reaching 0.86.