Stochastic Game Based Cooperative Alternating Q-Learning Caching in Dynamic D2D Networks

Edge caching has become an effective solution to cope with the challenges brought by the massive content delivery in cellular networks. In device-to-device (D2D) enabled caching cellular networks with time-varying content popularity distribution and user terminal (UT) location, we model these dynamic networks as a stochastic game to design a cooperative cache placement policy. The cache placement reward of each UT is defined as the caching incentive minus the transmission power cost for content caching and sharing. We consider the long-term cache placement reward of all UTs in this stochastic game. In an effort to solve the stochastic game problem, we propose a multi-agent cooperative alternating Q-learning (CAQL) based cache placement algorithm. The caching control unit is defined to execute the proposed CAQL, in which, the cache placement policy of each UT is alternatively updated according to the stable policy of other UTs during the learning process, until the stable cache placement policy of all the UTs in the cell is obtained. We discuss the convergence and complexity of CAQL, which obtains the stable cache placement policy with low space complexity. Simulation results show that the proposed algorithm can effectively reduce the backhaul load and the average content access delay in dynamic networks.