Robust Decision Making for Autonomous Vehicles at Highway On-Ramps: A Constrained Adversarial Reinforcement Learning Approach

被引:59
作者
He, Xiangkun [1 ]
Lou, Baichuan [1 ]
Yang, Haohan [1 ]
Lv, Chen [1 ]
机构
[1] Nanyang Technol Univ, Sch Mech & Aerosp Engn, Singapore 639798, Singapore
基金
新加坡国家研究基金会;
关键词
Autonomous vehicles; ramp merging; robust decision making; reinforcement learning; adversarial attack; AUTOMATED VEHICLES; OPTIMIZATION; GAME; GO;
D O I
10.1109/TITS.2022.3229518
中图分类号
TU [建筑科学];
学科分类号
0813 ;
摘要
Reinforcement learning has demonstrated its potential in a series of challenging domains. However, many real-world decision making tasks involve unpredictable environmental changes or unavoidable perception errors that are often enough to mislead an agent into making suboptimal decisions and even cause catastrophic failures. In light of these potential risks, reinforcement learning with application in safety-critical autonomous driving domain remains tricky without ensuring robustness against environmental uncertainties (e.g., road adhesion changes or measurement noises). Therefore, this paper proposes a novel constrained adversarial reinforcement learning approach for robust decision making of autonomous vehicles at highway on-ramps. Environmental disturbance is modelled as an adversarial agent that can learn an optimal adversarial policy to thwart the autonomous driving agent. Meanwhile, observation perturbation is approximated to maximize the variation of the perturbed policy through a white-box adversarial attack technique. Furthermore, a constrained adversarial actor-critic algorithm is presented to optimize an on-ramp merging policy while keeping the variations of the attacked driving policy and action-value function within bounds. Finally, the proposed robust highway on-ramp merging decision making method of autonomous vehicles is evaluated in three stochastic mixed traffic flows with different densities, and its effectiveness is demonstrated in comparison with the competitive baselines.
引用
收藏
页码:4103 / 4113
页数:11
相关论文
共 47 条
[41]   Harmonious Lane Changing via Deep Reinforcement Learning [J].
Wang, Guan ;
Hu, Jianming ;
Li, Zhiheng ;
Li, Li .
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, 2022, 23 (05) :4642-4650
[42]   Interpretable Decision-Making for Autonomous Vehicles at Highway On-Ramps With Latent Space Reinforcement Learning [J].
Wang, Huanjie ;
Gao, Hongbo ;
Yuan, Shihua ;
Zhao, Hongfei ;
Wang, Kelong ;
Wang, Xiulai ;
Li, Keqiang ;
Li, Deyi .
IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, 2021, 70 (09) :8707-8719
[43]  
Wang P, 2017, IEEE INT C INTELL TR
[44]   Outracing champion Gran Turismo drivers with deep reinforcement learning [J].
Wurman, Peter R. ;
Barrett, Samuel ;
Kawamoto, Kenta ;
MacGlashan, James ;
Subramanian, Kaushik ;
Walsh, Thomas J. ;
Capobianco, Roberto ;
Devlic, Alisa ;
Eckert, Franziska ;
Fuchs, Florian ;
Gilpin, Leilani ;
Khandelwal, Piyush ;
Kompella, Varun ;
Lin, HaoChih ;
MacAlpine, Patrick ;
Oller, Declan ;
Seno, Takuma ;
Sherstan, Craig ;
Thomure, Michael D. ;
Aghabozorgi, Houmehr ;
Barrett, Leon ;
Douglas, Rory ;
Whitehead, Dion ;
Duerr, Peter ;
Stone, Peter ;
Spranger, Michael ;
Kitano, Hiroaki .
NATURE, 2022, 602 (7896) :223-+
[45]   Automated vehicle's behavior decision making using deep reinforcement learning and high-fidelity simulation environment [J].
Ye, Yingjun ;
Zhang, Xiaohui ;
Sun, Jian .
TRANSPORTATION RESEARCH PART C-EMERGING TECHNOLOGIES, 2019, 107 :155-170
[46]   Adaptive Decision-Making for Automated Vehicles Under Roundabout Scenarios Using Optimization Embedded Reinforcement Learning [J].
Zhang, Yuxiang ;
Gao, Bingzhao ;
Guo, Lulu ;
Guo, Hongyan ;
Chen, Hong .
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2021, 32 (12) :5526-5538
[47]  
Zhang Y, 2018, IEEE INT VEH SYM, P1251, DOI 10.1109/IVS.2018.8500630