A Joint Reinforcement-Learning Enabled Caching and Cross-Layer Network Code in F-RAN With D2D Communications

In this paper, we leverage reinforcement learning (RL) and cross-layer network coding (CLNC) for efficiently pre-fetching requested contents to the local caches and delivering these contents to requesting users in a downlink fog-radio access network (F-RAN) with device-to-device (D2D) communications. In the considered system, fog access points (F-APs) and cache-enabled D2D (CE-D2D) users are equipped with local caches that alleviate traffic burden at the fronthaul and facilitate rapid delivery of the users' contents. To this end, the CLNC scheme optimizes the coding decisions, transmission rates, and power levels of both F-APs and CE-D2D users, and RL scheme optimizes caching strategy. A joint content placement and delivery problem is formulated as an optimization problem with a goal to maximize system sum-rate. The problem is an NP-hard problem. To efficiently solve it, we first develop an innovative decentralized CLNC coalition formation (CLNC-CF) switch algorithm to obtain a stable solution for the content delivery problem, where F-APs and CE-D2D users utilize CLNC resource allocation. By considering statistics of channel and users' content request into account, we then develop a multi-agent RL algorithm for optimizing the content placement at both F-APs and CE-D2D users. Simulation results show that the proposed joint CLNC-CF-RL framework can effectively improve the sum-rate by up to 30%, 60%, and 150%, respectively, compared to: 1) an optimal uncoded algorithm, 2) a standard rate-aware-NC algorithm, and 3) a benchmark classical NC with network-layer optimization.