Nearly optimal control of nonlinear Markovian systems subject to weak and strong interactions

This paper develops hierarchical control schemes of nonlinear dynamic systems under Markovian disturbances subject to weak and strong interactions. The state space of the underlying Markov chain is decomposed into several groups of recurrent states and a group of transient states resulting in a singularly perturbed Markov chain formulation. Instead of finding the control of the original system directly, a limit problem that is much simpler to handle is derived. Using the optimal control of the limit system, a near optimal control policy is constructed for the original system. The asymptotic optimality of the constructed control is proved.