Deep Reinforcement Learning Based Beam Selection for Hybrid Beamforming and User Grouping in Massive MIMO-NOMA System

This paper presents a deep reinforcement learning-based beam-user selection and hybrid beamforming design for the multiuser massive multiple-input multiple-output (MIMO) non-orthogonal multiple access (NOMA) downlink systems. The conventional hybrid beamforming in massive MIMO provides multiple directional beams, but each beam serves only one user. The integration of NOMA with the massive MIMO enables power domain multiplexing within a beam, hence increasing the system capacity. In this paper, we first design a channel gain and correlation-based users grouping algorithm per beam, and then using the deep reinforcement learning-based beam selection, a beamspace orthogonal analog precoder is obtained. The deep Q-network consists of a main network and target network with Adam optimizer. Finally, optimal power is allocated to the users in each beam. Simulation results show that at transmit SNR of 10 dB, the proposed scheme provides a 42% increase in sum-rate and energy efficiency performance as compared to the state-of-the-art K-means users' grouping and Stable Matching-based beam selection NOMA scheme.