Adaptive Optimal Control of Unknown Nonlinear Systems via Homotopy-Based Policy Iteration

As one efficient technique in reinforcement learning, policy iteration (PI) requires an initial admissible (or stabilizing for linear systems) control policy that renders the existing PI-based results to be model dependent. To attain a completely data-driven adaptive optimal control, this article suggests integrating a homotopic design with PI for unknown continuous-time nonlinear systems. Technically, we leverage a homotopic constant to construct an artificially stable system that allows zero control to initialize PI. Utilizing a homotopic strategy, we recursively update the artificial system and then enforce it to gradually recover the original system. This ultimately allows us to obtain an admissible control policy in a finite number of iterations without carrying out a model-based initialization. Once the admissible control is obtained, the proposed homotopic PI inherits fast convergence from the traditional PI technique and ensures learning the optimal control solution from the data measured from unknown nonlinear systems.