The Reinforcement Learning Approaches for Intelligent Collective System: A Survey

Intelligent Collective System（ICS）is an essential branch of artificial intelligence，encompassing various intelligent components that collectively give rise to an emergent phenomenon known as Collective Intelligence （CI）. CI exhibits the characteristics of self-organization in individual excitation， strong robustness in swarm convergence， and other characteristics. Based on ICS，AI enables the emergence of CI，providing a powerful framework for harnessing the potential of intelligent systems. Specifically， the decision-making process of ICS is a multifaceted and intricate nonlinear problem that intricately integrates humans，machines，and objects. This process spans across diverse spaces and encompasses various stages，including perception，decision-making，feedback，and optimization，forming a dynamic loop of information flow. Within this intricate framework， there exist abundant decision models that enable the system to consider a wide range of possibilities and alternatives. The traditional algorithms mainly rely on a large amount of knowledge and experience，creating a significant challenge in supporting the development of the system. The reliance on vast amounts of explicit knowledge and predefined rules limits the system’s ability to adapt and evolve in dynamic and complex environments. As the system encounters new situations or scenarios，its performance may suffer due to the lack of flexibility and adaptability inherent in these traditional algorithms. Reinforcement Learning （RL） is a powerful and comprehensive approach that seamlessly integrates perception and decision-making within an end-to-end framework. RL exhibits a remarkable autonomous learning capability， enabling systems to improve their performance through iterative optimization driven by trial and error. In RL， the system interacts with its environment，receiving feedback in the form of rewards or penalties based on its actions. Through this iterative process，the system learns to navigate complex decision spaces by exploring different actions and evaluating their consequences. By optimizing its decision-making policies over time，RL enables the system to acquire knowledge and adapt its behavior to maximize long-term rewards. Recently， there has been a remarkable evolution in RL algorithms， spurred by inspiration from both biological swarm behavior and artificial intelligence. These advancements have not only expanded the scope of RL from solving single-agent decision-making problems but have also paved the way for addressing joint collaboration problems involving multiple agents. As a result，RL has emerged as a new and promising avenue for the convergence and emergence of CI. However，the application of ICS faces significant challenges when dealing with various tasks. These challenges arise due to the unique characteristics of ICS， including the spatio-temporal sensitivity of the perceptual environment，the high autonomy of individuals within the swarm，the complex and variable relationships among agents，and the multi-dimensional nature of task goals. Based on the decision-making process of ICS and the operation mechanism of RL， this paper introduces RL algorithms that specifically target the challenges posed by ICS， focusing on four key aspects：joint communication， collaborative decision-making， reward feedback， and policy optimization. The paper further conducts an analysis of typical applications of RL algorithms in ICS，accompanied by a compilation of relevant open-source platforms and applicable algorithms. Finally，the paper addresses future research directions based on practical requirements. © 2023 Science Press. All rights reserved.