Survey of Deep Reinforcement Learning Methods with Evolutionary Algorithms

Deep reinforcement learning is one of the most important branches in the field of machine learning, which can achieve end-to-end learning through direct interaction with the environment and is capable of solving high-dimensional and large-scale problems. Although deep reinforcement learning has achieved remarkable results, it still faces problems such as insufficient exploration of the environment, poor robustness, and susceptibility of gradients caused by deceptive rewards. In general, evolutionary algorithms have good global search ability, robustness, parallelism and other advantages. Therefore, the methods combining evolutionary algorithms with deep reinforcement learning to compensate the inadequacy of deep reinforcement learning methods have become a research hotspot recently. This paper focuses on the applications of evolutionary algorithms in model-free deep reinforcement learning methods. We introduce evolutionary algorithms and basic methods of reinforcement learning firstly. After that, we introduce the characteristics, advantages, disadvantages, and applicable tasks of evolutionary algorithms, deep reinforcement learning algorithms, and combined methods of evolutionary algorithms and deep reinforcement learning, showing the necessity of combined methods from a different aspect. Then, two types of reinforcement learning methods with evolutionary algorithms are elaborated, which are reinforcement learning with evolutionary algorithms guided policy search and combination of evolutionary algorithms and deep reinforcement learning. In reinforcement learning with evolutionary algorithms guided policy search methods, we categorize the different policy search methods into parameter distribution search methods, policy gradient approximation methods, and policy population search methods. Parameter distribution search methods regard the parameters of a policy as a distribution and sample the parameters from this distribution to form a new policy. Policy gradient approximation methods use the fitness of the policy as an approximation of the gradient to update the parameters. Policy population search methods search directly from individuals in the policy population and select the individual with higher fitness. Then, we focus on the combined methods of evolutionary algorithms and deep reinforcement learning which attracts the interest of scholars currently, including evolutionary algorithm experience-guided deep reinforcement learning methods and evolutionary algorithm modules-embedded deep reinforcement learning methods. The evolutionary algorithm experience-guided deep reinforcement learning methods use experience obtained from individuals by continually interacting with the environment to guide the value network of reinforcement learning, while the evolutionary algorithm module-embedded deep reinforcement learning methods embed the evolutionary algorithm as an auxiliary module in the learning process of reinforcement learning. Furthermore, we compare and analyze these methods in detail. In particular, we compare the characteristics of various algorithms in the methods combing evolutionary algorithms and deep reinforcement learning, including without-feedback guidance methods and with-feedback guidance methods. We also compare the performance of various widely-used algorithms in with-feedback guidance methods on the continuous control tasks of MuJoCo and give a detailed analysis and future directions for improvement and research. Finally, we summarize all the combined methods of evolutionary algorithms and deep reinforcement learning mentioned in the paper, and we study the research emphasis and development trend of this field. Although evolutionary deep reinforcement learning frameworks have been proposed, we think these methods still require further theoretical study to balance the issues of exploration and exploitation. © 2022, Science Press. All right reserved.