Distributed Formation Coordinated Control of Multiple Unmanned Surface Vehicles Based on Deep Reinforcement Learning

Over the last decade, Unmanned Surface Vehicles (USVs) have gained significant traction in marine scientific research and military domains. Given the necessity for extensive coverage in many missions, the deployment of multiple USVs for coordinated operations has emerged as a viable strategy. This study presents a novel model-free deep reinforcement learning (DRL) approach for coordinating multiple USVs, with a particular emphasis on ensuring formation stability. First, a dynamic model for formation navigation in a non-stationary stochastic ocean environment is derived, accounting for both acceleration and angular velocity control. Next, a hierarchical leader-follower architecture is designed, facilitating the formation of stable chain formations and simplifying the control challenge. To address the complex, nonlinear coupling issues, deep reinforcement learning methods are employed, specifically the deep deterministic policy gradient (DDPG) and double deep Q-network (DDQN), both utilizing deep neural network (DNN) approximations. The comparative analysis of these methods in simulation showcases the effectiveness of the proposed control strategy and the adaptability of USVs in maintaining formation navigation even in non-stationary random environments.