Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control

被引:7
作者
Alegre, Lucas N. [1 ]
Bazzan, Ana L. C. [1 ]
da Silva, Bruno C. [2 ]
机构
[1] Univ Fed Rio Grande do Sul, Inst Informat, Porto Alegre, RS, Brazil
[2] Univ Massachusetts, CICS, Amherst, MA 01003 USA
关键词
Reinforcement learning; Traffic signal control; Non-stationarity; Multiagent systems;
D O I
10.7717/peerj-cs.575
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time and as a function of traffic control decisions taken in other parts of a network. In this paper we analyze the effects that different sources of non-stationarity have in a network of traffic signals, in which each signal is modeled as a learning agent. More precisely, we study both the effects of changing the context in which an agent learns (e.g., a change in flow rates experienced by it), as well as the effects of reducing agent observability of the true environment state. Partial observability may cause distinct states (in which distinct actions are optimal) to be seen as the same by the traffic signal agents. This, in turn, may lead to sub-optimal performance. We show that the lack of suitable sensors to provide a representative observation of the real state seems to affect the performance more drastically than the changes to the underlying traffic patterns.
引用
收藏
页数:20
相关论文
共 31 条
[21]  
Mannion P, 2016, AUTON SYST, P47, DOI 10.1007/978-3-319-25808-9_4
[22]  
National Research Council, HIGHWAY CAPACITY MAN
[23]  
Padakandla S., 2019, REINFORCEMENT LEARNI
[24]  
Padakandla Sindhu., 2020, A survey of Reinforcement Learning Algorithms for Dynamically Varying Environments
[25]  
Sutton Richard S., 2015, Reinforcement Learning: An Introduction
[26]  
Tan M., 1993, P 10 INT C MACH LEAR, V337, P330, DOI DOI 10.1016/B978-1-55860-307-3.50049-6
[27]  
TSITSIKLIS JN, 1994, MACH LEARN, V16, P185, DOI 10.1007/BF00993306
[28]  
Van der Pol E., 2016, P LEARN INFERENCE CO
[29]  
Wei H, 2019, SURVEY TRAFFIC SIGNA
[30]   A Survey on Reinforcement Learning Models and Algorithms for Traffic Signal Control [J].
Yau, Kok-Lim Alvin ;
Qadir, Junaid ;
Khoo, Hooi Ling ;
Ling, Mee Hong ;
Komisarczuk, Peter .
ACM COMPUTING SURVEYS, 2017, 50 (03)