A Model-free Framework for Nonlinear Optimal Control Based on Solving HJB Equation

This paper proposes a learning-based framework for searching the optimal feedback controller of a class of nonlinear optimal control problem based on solving HJB equation. By introducing Gaussian control noise for exploration, we aim to learn a robust controller in this stochastic case. This reformulation sacrifices some optimality to some extent, but as suggested in reinforcement learning (RL) exploration noise is essential to enable the model-free learning. The new stochastic optimal control problem is solved by finding the solution to the HJB equation, and is further reformulated to a constrained optimization problem that can be carried out by a model-free algorithm. Compared with the Markovian framework of RL, our method is derived under stochastic differential equation description in continuous time and space, thus is preferred from a theoretical point of view. We demonstrate the practical potential of the proposed method by three classical nonlinear control tasks.