ERGODIC THEORY FOR CONTROLLED MARKOV CHAINS WITH STATIONARY INPUTS

Consider a stochastic process X on a finite state space X = {1,...,}d. It is conditionally Markov, given a real-valued "input process" sigma.0 This is assumed to be small, which is modeled through the scaling, sigma(t) = epsilon sigma(1)(t), 0 <= epsilon <= 1, where sigma(1) is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on sigma : (i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain X obtained with sigma equivalent to 0. The triple (X, X-center dot, sigma) is a jointly stationary process satisfying P{X(t) not equal X-center dot(t)} = 0(epsilon). Moreover, a second-order Taylor-series approximation is obtained: P{X(t) = P{X-center dot(t)= i} + epsilon(2)pi((2))(i) + 0(epsilon(2)), 1 <= i <= d, with an explicit formula for the vector n ((2)) is an element of R-d. (ii) For any m >= 1 and any function f : {1,...,d} x R -> R-m, the stationary stochastic process Y (t) = f (X (t), sigma(t)) has a power spectral density S-f that admits a second-order Taylor series expansion: A function) S-f((2)) : [-pi, pi] -> C-mxm is constructed such that S-f (theta) = S-f(center dot) (theta) epsilon(2)sf(2)(theta) + o(epsilon(2)), theta is an element of[-pi,pi] in which the first term is the power spectral density obtained with epsilon = 0. An explicit formula for the function S-f((2)) is obtained, based in part on the bounds in (i). The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.