Hierarchical Reinforcement Learning for Relay Selection and Power Optimization in Two-Hop Cooperative Relay Network

In this paper, we study the outage probability minimizing problem in a two-hop cooperative relay network. To reduce outage probability, existing studies propose many schemes for relay selection and power allocation, which are usually based on the assumption of exact channel state information (CSI). However, it is difficult to obtain perfect instantaneous CSI in practical situations where channel states change rapidly, and thus traditional methods would not perform well. Considering these factors, we turn to the emerging reinforcement learning (RL) methods for solutions. RL methods do not need any prior knowledge of CSI, but use neural network for approximation and decision after interacting with communication environment. Nevertheless, conventional RL methods, including most deep reinforcement learning (DRL) methods, cannot perform well when the search space is too large. In addition, non-stationarity is a common problem when using hierarchical reinforcement learning (HRL), which is caused by the changing behavior in different hierarchies. Therefore, we first propose a DRL framework with an outage-based reward function, which is then used as a baseline. Then, we further design an HRL framework and training algorithm. By decomposing relay selection and power allocation into two hierarchical optimization objectives, and combining on- policy and off-policy methods in the HRL framework, our method successfully address the sparse reward and non-stationary problem. Simulation results reveal that compared with traditional DRL method, the proposed HRL training algorithm can converge faster and reduce the outage probability by 8% in two-hop relay network with the same outage threshold.