Fast Online Reinforcement Learning of Distributed Optimal Controller for Large-Scale Network Systems

In this paper, we propose a fast real-time reinforcement learning (RL) control algorithm to design distributed controllers for large-scale network systems. When network size is large, existing RL-based methodologies can result in unacceptably long learning time, making them unsuitable for real-time control. The proposed approach overcomes this issue by aggregating states while keeping the aggregation error as small as possible. The aggregation matrix is constructed by a kind of sparse singular value decomposition of data. Next, a distributed controller is learned using the aggregated data by the RL method which is modified to promote sparsity of the controller by l(1)-regularization. Because of the structure of the aggregation matrix, the resultant controller can have a highly sparse structure. The efficiency of the proposed method is shown through a numerical simulation of a complex network system whose graph structure is described by the Barabasi-albert model.