Deep Learning Based Joint Beam Selection and Precoding Design for mmWave Systems with Lens Arrays

In this work, we investigate the joint design of beam selection and digital precoding matrices for millimeter wave (mmWave) multiuser multiple-input multiple-output (MU-MIMO) systems with discrete lens arrays (DLA) to maximize the sum-rate. To tackle this challenging problem with discrete variables and coupled constraints, we propose an efficient framework of joint neural network (NN) design. Specifically, the proposed framework consists of a deep reinforcement learning (DRL)-based NN and a deep-unfolding NN, which are employed to optimize the beam selection and digital precoding matrices, respectively. As for the DRL-based NN, we formulate the beam selection problem as a Markov decision process and a double deep Q-network algorithm is developed to solve it. Regarding the design of the digital precoding matrix, we develop an iterative weighted minimum mean-square error algorithm induced deep-unfolding NN, which unfolds this algorithm into a layer-wise structure. Simulation results show that our proposed jointly trained NN significantly outperforms the existing iterative algorithms.