Deep Reinforcement Learning for Age of Information Minimization in Reservation Multi-Access Networks

This paper aims to optimize age of information (AoI)-enabled user scheduling in reservation multi-access networks through the application of deep reinforcement learning (DRL). In such networks, frames comprise reservation slots and multiple data slots, with reservation slots further divided into mini-slots. Users vying to transmit status updates randomly send reservation packets within these mini-slots, competing for data slots in the current frame. Collisions at the access point (AP) arise when multiple users reserve the same mini-slot, resulting in heightened network AoI. We formulate the AoI-enabled user scheduling problem as a Markov decision problem (MDP), solvable through DRL algorithms. To address challenges like slow convergence and local optimality typical in traditional reinforcement learning, we employ soft actor-critic (SAC). In collisions within a mini-slot, the AP, acting as the agent, selects the most suitable user and allocates the corresponding data slot for packet transmission, enhancing information freshness in reservation multi-access networks. The agent effectively converges to optimal solutions while remaining robust to discrepancies between training and actual environments by optimizing expected returns and policy entropy. We validate the proposed scheme through implementation, evaluating its convergence, effectiveness, and robustness.