User Association in a VHetNet With Delayed CSI: A Deep Reinforcement Learning Approach

Non-terrestrial base stations (NTBSs) must be employed for next-generation wireless networks to provide users with ubiquitous connectivity and a higher data rate. In vertical heterogeneous networks (VHetNets), associating users with either a terrestrial base station (TBS) or a NTBS to maximize the sum-rate of the network while accounting for the resource limitations that exist at the NTBS poses a challenge. Moreover, a practical user association method should be capable of working in a realistic situation in which instantaneous channel state information (CSI) is not available. To solve this problem, we propose a deep Q-learning (DQL) approach in which a satellite is our agent and schedules each user to a TBS or a high-altitude platform station (HAPS) in each time slot using the CSI of the previous time slot. The proposed method achieves nearly identical results as the exhaustive search action selection method. Furthermore, we investigate the effect of imperfect CSI and show our proposed method outperforms the convex optimization user association scheme in the presence of noisy CSI.