Enhancing Autonomous Lane-Changing Safety: Deep Reinforcement Learning via Pre-Exploration in Parallel Imaginary Environments

The connected and autonomous vehicles combined with deep reinforcement learning (DRL) are capable of handling complex driving scenarios. However, due to the random exploration feature of reinforcement learning (RL), unexpected actions and collisions that would be inevitable in the real world occur during training, resulting in property damage, injury, and loss of life. To address this issue, in this article, we propose a sophisticated safe DRL in autonomous lane changing that benefits from both exploration and optimization capabilities. The key idea is first to integrate the safety constraints into the RL algorithm to limit the actions that the agent can take during training, which is implemented by designing a vehicle convex occupancy approximation to estimate the candidate action set. Then, adaptive exploration strategies are used, in which an imaginary environment based on domain randomization is built to explore areas of the action-state space where it is uncertain about the outcomes. We present a Monte Carlo tree search to replace unsafe with safe action. The twin-delayed deep deterministic policy gradient is used as the RL algorithm to train action space agents. Experimental results show that our proposed framework significantly enhances safety during the lane-change process with faster and more stable learning than the other methods.