Input-Decoupled Q-Learning for Optimal Control

A design of optimal controllers based on a reinforcement learning method called Q-Learning is presented. Central to Q-Learning is the Q-function which is a function of the state and all input variables. This paper shows that decoupled-in-the-inputs Q-functions exist, and can be used to find the optimal controllers for each input individually. The method thus converts a multiple-variable optimization problem into much simpler single-variable optimization problems while achieving optimality. An explicit model of the system is not required to learn these decoupled Q-functions, but rather the method relies on the ability to probe the system and observe its state transition. Derived within the framework of modern control theory, the method is applicable to both linear and non-linear systems.