MEAN-FIELD MARKOV DECISION PROCESSES WITH COMMON NOISE AND OPEN-LOOP CONTROLS

被引:14
作者
Motte, Mederic [1 ]
Huyen Pham [1 ]
机构
[1] Univ Paris, LPSM, Paris, France
关键词
Mean-field; Markov decision process; conditional propagation of chaos; measurable coupling; randomized control; CONVERGENCE;
D O I
10.1214/21-AAP1713
中图分类号
O21 [概率论与数理统计]; C8 [统计学];
学科分类号
020208 ; 070103 ; 0714 ;
摘要
We develop an exhaustive study of Markov decision process (MDP) under mean field interaction both on states and actions in the presence of common noise, and when optimization is performed over open-loop controls on infinite horizon. Such model, called CMKV-MDP for conditional McKean- Vlasov MDP, arises and is obtained here rigorously with a rate of convergence as the asymptotic problem of N-cooperative agents controlled by a social planner/influencer that observes the environment noises but not necessarily the individual states of the agents. We highlight the crucial role of relaxed controls and randomization hypothesis for this class of models with respect to classical MDP theory. We prove the correspondence between CMKV-MDP and a general lifted MDP on the space of probability measures, and establish the dynamic programming Bellman fixed point equation satisfied by the value function, as well as the existence of e -optimal randomized feedback controls. The arguments of proof involve an original measurable optimal coupling for the Wasserstein distance. This provides a procedure for learning strategies in a large population of interacting collaborative agents.
引用
收藏
页码:1421 / 1458
页数:38
相关论文
共 39 条