Reinforcement Learning for Stochastic Max-Plus Linear Systems

This paper studies the design of control policies for Discrete Event Systems under uncertainties. We capture the timing of the events using the framework of max-plus-linear systems in which the time between consecutive events depends on random delays with unknown distributions. Our policy synthesis approach is with respect to a cost function, and it can be extended directly to satisfy safety specifications on the timing of events. The main novelty of our approach is to translate the system evolution to a Markov decision process (MDP) that has an uncountable state space and develop a stochastic optimisation problem under the evolution of the MDP. To tackle the unknown distribution of uncertainties (thus unknown transition probabilities in the MDP), we employ model-free reinforcement learning to perform optimisations and find control policies for the system. Our implementation results on the 9-dimensional model of a railway network show superiority of our learning approach in comparison with the stochastic model predictive control approach.