Joint Optimization of Caching, Computing, and Radio Resources for Fog-Enabled IoT Using Natural Actor-Critic Deep Reinforcement Learning

The cloud-based Internet of Things (IoT) develops rapidly but suffer from large latency and backhaul bandwidth requirement, the technology of fog computing and caching has emerged as a promising paradigm for IoT to provide proximity services, and thus reduce service latency and save backhaul bandwidth. However, the performance of the fog-enabled IoT depends on the intelligent and efficient management of various network resources, and consequently the synergy of caching, computing, and communications becomes the big challenge. This paper simultaneously tackles the issues of content caching strategy, computation offloading policy, and radio resource allocation, and propose a joint optimization solution for the fog-enabled IoT. Since wireless signals and service requests have stochastic properties, we use the actor-critic reinforcement learning framework to solve the joint decision-making problem with the objective of minimizing the average end-to-end delay. The deep neural network (DNN) is employed as the function approximator to estimate the value functions in the critic part due to the extremely large state and action space in our problem. The actor part uses another DNN to represent a parameterized stochastic policy and improves the policy with the help of the critic. Furthermore, the Natural policy gradient method is used to avoid converging to the local maximum. Using the numerical simulations, we demonstrate the learning capacity of the proposed algorithm and analyze the end-to-end service latency.