Suboptimal reduced control of unknown nonlinear singularly perturbed systems via reinforcement learning

In this paper, a suboptimal reduced control method is proposed for a class of nonlinear singularly perturbed systems (SPSs) with unknown dynamics. By using singular perturbation theory, the original system is reduced to a reduced system, by which a policy iterative method is proposed to solve the corresponding reduced Hamilton-Jacobi-Bellman (HJB) equation with convergence guaranteed. A reinforcement learning (RL) algorithm is proposed to implement the policy iterative method without using any knowledge of the system dynamics. In the RL algorithm, the unmeasurable state of the virtual reduced system is reconstructed by the slow state measurements of the original system, the controller and cost function are approximated by actor-critic neural networks (NNs) and the method of weighted residuals is utilized to update the NN weights. The influence introduced by state reconstruction error and NN function approximation on the convergence, suboptimality of the reduced controller and stability of the closed-loop SPSs are rigorously analyzed. Finally, the effectiveness of our proposed method is illustrated by examples.