On the effect of probing noise in optimal control LQR via Q-learning using adaptive filtering algorithms

The probing noise is essential to obtain the persistence of excitation condition for solving Bellman equations in optimal control problems based on reinforcement learning. However, the level of probing noise affects the system state and the input control signal. The Bellman equation solution can be obtained using the classical adaptive filtering algorithms, namely, normalized least-mean-square (NLMS) and recursive-least-squares (RLS); thus, taking into account this kind of solution, in this paper, we present an analysis on the effect of probing noise in the state and input of the dynamic system, considering the linear quadratic regulator (LQR) problem using Q-learning policy iteration. In our analysis, a closed formula for the autocovariance matrices of the system state and the input are obtained, showing that their norms are proportional to the variance of the probing noise during the learning process. Numerical experiments show the performance of Q-learning policy iteration method based on NLMS and RLS for different variances of probing noise. (c) 2022 European Control Association. Published by Elsevier Ltd. All rights reserved.