DTSRL: Efficient reinforcement learning for approximate optimal tracking control of discrete-time nonlinear systems

The optimal tracking control for unknown nonlinear systems via reinforcement learning (RL) is still an open problem in the absence of an initial stabilizing policy. Its fundamental challenge lies in improving poor learning efficiency, arising from an additional switching term. In view of such a difficulty, this paper proposes a dual-time-scale RL (DTSRL) algorithm by devising an alternate learning mechanism aiming to enhance learning efficiency. Specifically, a convergence boundary factor is introduced into update of the actor-critic networks to establish an updating regulator, thereby relaxing the switching condition. Then, an alternate learning mechanism is devised to achieve a steady switching procedure by integrating this updating regulator into a dual-time-scale manner, effectively mitigating the impact of approximation errors. In addition, the tracking error and the weight estimation errors of the actor-critic networks are demonstrated to be uniformly ultimately bounded by means of the Lyapunov method. A comparative simulation is carried out to show that our DTSRL outperforms the existing works in terms of the learning efficiency.