Trajectory and Communication Design for Cache- Enabled UAVs in Cellular Networks: A Deep Reinforcement Learning Approach

In this article, we investigate the content transmission in a heavy-crowded multiple access cellular network, whose data traffic is offloaded through the combination of edge caching and unmanned aerial vehicle (UAV) communication. In this context, we formulate a novel optimization problem, which minimizes the sum content acquisition delay of users by optimizing the multiuser association and cache placement jointly with UAV trajectory and transmission power over a given flight duration. However, due to the uncertainty of the environment (e.g., random content requests and dynamic UAV positions), it is often difficult and impractical to solve the formulated problem using conventional optimization methods. To this end, we model our problem as a partially observable stochastic game where the macro base station (MBS) and UAVs act as agents to collectively interact with the environment to receive distinctive observations. Moreover, we take advantage of the Proximal Policy Optimization (PPO) learning strategy and propose a novel Dual-Clip PPO-based algorithm to solve the converted problem. To guide agent exploration, a new exploration criterion is proposed in which each UAV agent can obtain an intrinsic reward when it explores beyond the boundary of explored regions (BeBold). Note that the MBS agent has the extrinsic reward given by the environment only. Numerical results reveal that the proposed algorithm outperforms the standard PPO-based deep reinforcement learning algorithm. Moreover, the proposed joint design scheme can achieve a dramatic reduction of content acquisition delay compared with the benchmark schemes.