Tacit Commitments Emergence in Multi-agent Reinforcement Learning

Tacit commitments have been widely seen as a crucial underpinning for real-world cooperation. Similarly, it could also be a key to multi-agent cooperation. This paper proposes a novel tacit commitment emergence multi-agent reinforcement learning (MARL) framework (TCEM). In MARL, we define commitment as the unique state that the agent will exhibit through its action. TCEM first equips each agent with a commitment inference module (CIM) to infer its neighbor's commitments. Then, TCEM proposes that commitments influence intrinsic motivation (CIR) to encourage agents to have casual influence on others' actions. Finally, commitment acceptance intrinsic (CAI) motivation is constructed to guide the agent in behaving considering neighbors' commitments. CIR and CAI calculate intrinsic reward using counterfactual reasoning deriving from causal inference. Empirical results show that our method can effectively improve learning performance and deliver better cooperation among agents, which helps our method show superior performance on the Google Research Football benchmark.