Reward-Filtering-Based Credit Assignment for Multi-Agent Deep Reinforcement Learning

In recent decades, reinforcement learning has achieved remarkable successes in many fields such as intelligent traffic control, competitive gaming, unmanned system positioning, and navigation. As more and more realistic scenarios require multi-agent to undertake complex tasks cooperatively, researchers pay more attention to studying multi-agent than single agents in reinforcement learning. At the same time, in multi-agent reinforcement learning (MARL), learning cooperation is a new research hotspot, which means agents need to learn to cooperate using only actions and local observations. However, the credit assignment problem needs to be solved when studying the cooperative process of a multi-agent system with DRL. In the process of learning to complete tasks, the partially observable environment provides reward reinforcement signals for the joint actions produced by agents, which are used to update the parameters of the deep reinforcement learning network. But the global reward is non-Markovian. When an agent takes action at the current state, the actual reward signal for this action is usually given after several time steps. Especially in a difficult multi-agent environment, this phenomenon is more serious. In addition, all agents share the same global reward, making it hard to determine how much each agent in the system contributes to the whole. When an agent learns the strategy well in advance and gains a high return, the others may stop exploring, which leads the whole system to trap in the local optimum. To solve these problems, this paper introduces a credit assignment algorithm based on reward filtering that is not restricted by action space. The goal is to restore the local reward of each agent from the global reward obtained by all agents and apply it to the training of the action-value function network. The exploration behaviour of other agents often causes the non-stationarity of the environment, and the agent's own reward signal can be obtained by removing the influence of non-stationary from the global reward. Based on this, starting from the global reward, we model the influence caused by non-stationary factors as noise and propose a reward filter-based estimation mechanism to update the value function. In the process of centralized training, the influence of other agents on the environment is modelled as noise. The local reward of each agent is obtained by filtering the global reward, which is used to coordinate the behaviours of multi-agents and improve the system reward. We also propose a multi-agent deep reinforcement learning framework based on reward filtering (RF-MADRL) and successfully validate it in cooperative and competitive environments, namely the cooperative navigation with obstacles and predator-prey environments of Open AI. The experimental results show that compared with the baseline methods, including the traditional MADDPG method, value-function-based algorithms (i.e., VDN, QMIX, and QTRAN) and actor-critic-based credit assignment algorithms (i.e., COMA, FacMADDPG, and MAAC), our proposed RFMADRL has a better performance. The policy convergence is faster, and the reward obtained by the agent system is higher. The ablation experiment analysis shows that the reward filter module effectively improves the agent system's reward and solves the credit assignment problem. © 2022, Science Press. All right reserved.