Deep Learning-Assisted Signal Detection for OTFS-NOMA Systems

Orthogonal time frequency space (OTFS) modulation is introduced as a modulation technique known for its strong performance in high-Doppler scenarios. This two-dimensional modulation method involves multiplexing information symbols in the delay-Doppler (DD) domain. This study presents a deep learning (DL) based signal detection for OTFS non-orthogonal multiple access (NOMA) communication networks. In this work, the OTFS known as a popular sixth-generation (6G) candidate solution with enhanced spectral efficiency in high-mobility environments, is combined with NOMA over Rayleigh fading channels. In addition, a DL-based signal detection approach for the OTFS-NOMA scheme is proposed, where the network is trained to distinguish and decode the signals effectively. This enhances the overall system performance and paves the way for more efficient and reliable communication in high-mobility wireless environments. In our study, signal recovery employs a bidirectional long short-term memory (BiLSTM) network. The comparison of the message passing (MP) algorithm and the BiLSTM technique regarding symbol error rate (SER) performance for detecting signals over near and far users is evaluated. Furthermore, we examine the impact of the three common optimizers on the SER achievement for training optimizer selection. Moreover, the numerical results show that the root mean squared propagation (RMSprop) outperforms the other optimizer selection techniques regarding SER. Finally, the performance of the BiLSTM technique is observed to be better than that of the MP, except for the stochastic gradient descent (SGD) optimizer. RMSprop and the adaptive momentum optimizer (Adam) yield a maximum training accuracy of 99.9%.