Constrained Reinforcement Learning-Based Closed-Loop Reference Model for Optimal Tracking Control of Unknown Continuous-Time Systems

Although reinforcement learning (RL) is effective in stabilizing systems, it faces many challenges in solving the tracking problem of unknown continuous-time systems. One of the major challenges is that RL-based control can hardly satisfy both the transient and steady-state performance requirements for the tracking problem simultaneously. In this study, instead of implementing an RL controller, the RL agent acts as a planner in the closed-loop reference model. The RL-based planner concentrates on tracking performance optimization by the constrained integral RL algorithm. Meanwhile, the system is controlled by the proposed library-based adaptive controller, which contains a library of candidate functions for modeling the unknown system dynamics. A natural gradient-like adaptive law is developed to update the controller, ensuring asymptotic tracking and promoting sparsity in the controller parameter. Compared with the conventional RL-based control, the proposed framework can eliminate the tracking error while avoiding the high-frequency oscillation and peaking phenomenon. Furthermore, we theoretically demonstrate that our approach can improve the transient performance in terms of the L-2 norm of the tracking error and explicitly limit the L-infinity norm of the peaking value through the Lyapunov analysis. Simulations are presented to support the theoretical findings at the end of the paper.