Towards Network Dynamics: Adaptive Buffer Management with Deep Reinforcement Learning

The prosperity of cloud computing and 5G/B5G is bringing a wide range of delay jitter-sensitive applications (e.g., professional audio/video streaming and industrial automation) to large-scale IP networks. Although various network-side techniques have been proposed to guarantee the quality of service (QoS), the client-side technique by introducing a receive buffer should never be neglected from the applications' perspective. In this paper, we revisit the buffer management problem to address the disadvantages of state-of-the-art studies, which assumed network characteristics known a prior with simplified or inaccurate network models and failed to adapt to network dynamics. Specifically, we propose adaptive buffer management with deep reinforcement learning, i.e., DRL-ABM. We first define a tradeoff value to measure the buffer management performance in terms of the start-up delay, underflow frequency and packet losses. Then we formulate the DRL model and design deep neural networks (DNNs) based on the advantage actor critic (A2C) algorithm. To evaluate the performance of DRL-ABM, we perform extensive simulations. Simulation results show that DRLABM can achieve better buffer management performance, i.e., reducing the tradeoff value by at least 20% when compared with the benchmarks TBM and ABM. Moreover, DRL-ABM reduces packet losses to approximately 0, indicating that a smaller receive buffer is sufficient if managed with DRL-ABM.