Distributed Convolutional Deep Reinforcement Learning based OFDMA MAC for 802.11ax

The IEEE 802.11ax also known as Wi-Fi 6, incorporates multi-user (MU) Orthogonal Frequency Division Multiple Access (OFDMA) based distributed up-link communication, in which stations obtain packet transmission opportunities in accordance with the OFDMA back-off (OBO) procedure and then randomly select one of the available sub-channel, called Resource Unit (RU). However, this random RU selection lead to a high collision rate, consequently degrading throughput and increasing latency. This paper proposes a distributed RU selection method using Convolutional Neural Network (CNN) based Deep Reinforcement Learning (C-DRL) to improve throughput and latency of a wireless network. Specifically, each station locally trains its CNN in an online manner on the basis of energy detection and acknowledgment packets. To reach a steady-state faster, we propose the Greedy Experience Replay (GER) algorithm, in which stations also learn from the non-selected RUs by hypothetically generating their outcomes in hindsight. Notably, the C-DRL RU selection does not require any centralized training or packet exchanges amongst the stations to ensure fair resource distribution. Further C-DRL stations can coexist with standard 802.11ax stations and still improve overall network performance. Comprehensive simulations were conducted to demonstrate the performance of the proposed C-DRL method. The results show a 112.7% higher average throughput and 73.5% lower average latency than standard 802.11ax medium access control (MAC) for a single access point network (ten stations).