Heuristic-Based Multi-Agent Deep Reinforcement Learning Approach for Coordinating Connected and Automated Vehicles at Non-Signalized Intersection

One typical application of connected and automated vehicles (CAVs) is to coordinate multiple CAVs at a non-signalized intersection in mixed traffic, and it may take advantage of multi-agent deep reinforcement learning (MDRL) approaches to improve the overall coordination efficiency. This study proposes a heuristic-based MDRL algorithm (H-QMIX) developed based on a value-based MDRL algorithm, QMIX. This algorithm incorporates a heuristic-based action mask module to guide CAVs efficiently and safely through intersections, composed of a stimulative passing sequence and safety restrictions on CAVs' action space in the junction area. Compared with other MDRL algorithms (e.g., IPPO, QMIX), the H-QMIX algorithm demonstrates improved training performance in terms of safety and efficiency in two case studies, where the first requires all CAVs to affix their routes, and another allows CAVs to choose random routes. Concerning the model's generalization ability, the trained models with the maximal episodic return are then transferred to a more practical scenario with a certain vehicle-to-vehicle (V2V) communication delay in a zero-shot manner. The simulation results illustrate that H-QMIX is robust to a certain communication delay. The code for this paper is available at: https://github.com/flammingRaven/heuristic_based_qmix.