Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning

In the case that several controllable events (control commands) are allowed to execute simultaneously, the supervisor in the framework of discrete event systems (DESs) selects one randomly. However, in practical applications, such as traffic scheduling and robot path planning, the problems of directed control and numerical optimization should be considered. This paper introduces an optimization mechanism to quantify the control cost and combines supervisory control theory (SCT) with reinforcement learning. A systematic procedure is proposed to synthesize the optimal directed supervisor of a DES based on reinforcement learning, which makes the controlled system achieve the following three goals: (1) the control specifications relevant to security and liveness are not violated; (2) at most one controllable event can be executed at each state; (3) the cumulative cost of event execution from the initial state to a mark state is minimal. First, given the automaton models of the plant and specifications, the target automaton model is obtained by the synchronous operation of these two models; a cost function is defined and assigns the execution cost for each event in the target model. Second, the nonblocking and maximally permissive supervisor is synthesized by SCT. Finally, the supervisor is transformed into a Markov decision process and then the Q-learning algorithm is utilized to compute the optimal directed supervisor. Two applications are used to verify the effectiveness and correctness of the proposed method. The simulation results show that the proposed method can realize the directed control of the system, and the numerical cost of the directed supervisor is minimized. © 2024 Chinese Institute of Electronics. All rights reserved.