Unveiling the power of features: A comparative study of machine learning and deep learning for modulation recognition

Wireless communication systems rely on amplitude, frequency, and phase parameters for signal transmission. Traditional modulation recognition (MR) techniques, employing likelihood-based (LB) and feature-based (FB) methods, struggle with accurate classification, particularly at low signal-to-noise ratios (SNR) and increasing modulation complexity. Machine learning (ML) and deep learning (DL) algorithms, which efficiently utilize in-phase/quadrature (IQ) and r-radius/θ-angle (rθ) data representations to enhance MR performance. DL, utilizing artificial neural networks (ANN), minimizes the need for extensive feature engineering, making it adept at handling diverse modulation types and challenging SNR conditions. This study systematically examines dataset generation parameters to reveal their impact on MR performance. By focusing on these underlying parameters, the analysis provides deeper insights into how data characteristics influence model performance, offering a foundational understanding for optimizing dataset configurations in MR tasks. Evaluating ML and DL models across datasets, results show DL model consistently outperforms ML models, achieving up to 79.41 % accuracy on IQ-based datasets. DL's hierarchical feature extraction enhances adaptability, particularly with larger datasets, reduced window lengths (WL), and specific θ ranges (e.g., radians or smaller degree intervals). For ML models, datasets based on IQ, rθ, and IQrθ parameters yield better results but remain below 70 % accuracy. Overall, DL model exhibits robust adaptability to complex signal environments, highlighting their effectiveness in advancing modulation recognition for next-generation wireless communication systems. © 2025 Elsevier B.V.