Q-learning Based Adaptive Optimal Control for Linear Quadratic Tracking Problem

This paper describes a Q-learning based algorithm to design the linear quadratic tracker (LQT) for linear time invariant (LTI) continuous-time systems with partially unknown dynamics. The proposed approach uses a fixed-point equation in terms of Q-function in order to estimate the unknown optimal gain parameters. The fixedpoint equation, which is derived by applying the Pontryagin's minimum principle in Q-learning, is based on the modified algebraic Riccati equation (ARE) for LQT problem. The online adaptation of the optimal parameters are achieved by using the gradient descent based parameter update laws by minimizing the Bellman's error term which is derived from fixed-point equation mentioned earlier. A persistence of excitation condition has been used to establish the desired optimal convergence of the estimated control parameters. Simulation results have been shown to validate the efficiency of the proposed Q-learning approach.