Proximal policy optimization for formation navigation and obstacle avoidance

In this paper, a formation control problem of second-order holonomic agents is considered, where agents navigate around obstacles using proximal policy optimization (PPO)-based deep reinforcement learning (DRL). The formation is allowed to shrink and expand, while maintaining its shape, in order to navigate the geometric centroid of the formation towards the goal. A bearing-based reward function is presented that depends on the bearing error of each agent towards its designated neighbors. The agents share a single policy that is trained in a centralized manner. Distance measurements, state information, error information regarding neighboring agents, and simulation information are used for training the policy in an end-to-end fashion. Simulation results using the proposed approach are compared with that obtained using an angle-based reward function.