Survey of Deep Reinforcement Learning Based on Value Function and Policy Gradient

As a hot research problem in the field of artificial intelligence, Deep Reinforcement Learning(DRL) has attracted more and more attention since it was proposed. At present, DRL can solve many problems that were previously difficult to solve such as learning how to play video games directly from raw pixels and learning a control strategy for robot problems. DRL builds an autonomous system with a higher level understanding of the visual world by a continous optimization of the control strategy. Among them, DRL based on value function and policy gradient is the core basic method and research focus. This paper systematically elaborates and summarizes two types of DRL methods including solving algorithms and network structures. Firstly, DRL methods based on value function are summarized, including Deep Q-Network(DQN) and improved methods based on DQN. DQN is a pioneering work in the field of DRL. This model trains Convolutional Neural Network (CNN) with a variety of Q learning. Before the emergence of DQN, the problem of instability or even non-convergence will appear when the action value function in Reinforcement Learning(RL) is approximated by neural network. To solve this problem, DQN uses two technologies: the experience replay mechanism and the target network. According to different emphasis on DQN improvement, various improved versions based on DQN can be divided into four categories: improvement of training algorithm, improvement of neural network structure, improvement of introduction of new learning mechanism and improvement based on new proposed RL algorithm. The research motivation, overall thinking, advantages and disadvantages, application scope and performance of DQN improvement are elaborated in detail. Then the concept and common algorithms of policy gradient are introduced. Policy gradient algorithm is widely used for RL problems in continuous space. Its main idea is to parameterize the policy, calculate the policy gradient about the action and the action is adjusted continuously along the direction of the gradient and the optimal policy is gradually obtained. The common policy gradient algorithm includes REINFORCE algorithm and Actor-Critic algorithm. Also, DRL methods based on policy gradient are summarized including Deep Deterministic Policy Gradient(DDPG), Trust Region Policy Optimization(TRPO), Asynchronous Advantage Actor-Critic(A3C) and some corresponding improved methods. Drawing on DQN technology, DDPG adopts the experience replay mechanism and a separate target network to reduce the correlation between data and increase the stability and robustness of the algorithm. The problem solved by TRPO is to select the appropriate step size by introducing the trust region constraint defined by Kullback-Leibler divergence so as to ensure that the optimization of the policy is always in the good direction. A3C uses a conceptually simple and lightweight DRL framework and optimizes the deep neural network controller using an asynchronous gradient descent method. Then, AlphaGo and Alpha Zero which represent advanced research achievements of DRL are summarized and the relationship between the latter and the two DRL methods summarized in this paper is analyzed. Then some common experimental platforms of DRL algorithms are introduced including ALE, OpenAI Gym, RLLab, MuJoCo and TORCS. Finally, the future research directions of DRL are prospected. © 2019, Science Press. All right reserved.