Cooperative reinforcement learning in topology-based multi-agent systems

被引:8
作者
Xiao, Dan [1 ]
Tan, Ah-Hwee [1 ]
机构
[1] Nanyang Technol Univ, Sch Comp Engn, Singapore 639798, Singapore
关键词
Topology-based multi-agent systems; Cooperative learning; Reinforcement learning; Binary tree formation; Policy sharing; SUPPLY CHAIN; ALGORITHM; ARCHITECTURE;
D O I
10.1007/s10458-011-9183-4
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Topology-based multi-agent systems (TMAS), wherein agents interact with one another according to their spatial relationship in a network, are well suited for problems with topological constraints. In a TMAS system, however, each agent may have a different state space, which can be rather large. Consequently, traditional approaches to multi-agent cooperative learning may not be able to scale up with the complexity of the network topology. In this paper, we propose a cooperative learning strategy, under which autonomous agents are assembled in a binary tree formation (BTF). By constraining the interaction between agents, we effectively unify the state space of individual agents and enable policy sharing across agents. Our complexity analysis indicates that multi-agent systems with the BTF have a much smaller state space and a higher level of flexibility, compared with the general form of n-ary (n > 2) tree formation. We have applied the proposed cooperative learning strategy to a class of reinforcement learning agents known as temporal difference-fusion architecture for learning and cognition (TD-FALCON). Comparative experiments based on a generic network routing problem, which is a typical TMAS domain, show that the TD-FALCON BTF teams outperform alternative methods, including TD-FALCON teams in single agent and n-ary tree formation, a Q-learning method based on the table lookup mechanism, as well as a classical linear programming algorithm. Our study further shows that TD-FALCON BTF can adapt and function well under various scales of network complexity and traffic volume in TMAS domains.
引用
收藏
页码:86 / 119
页数:34
相关论文
共 50 条
[41]   Cooperative Action Acquisition Based on Intention Estimation in a Multi-Agent Reinforcement Learning System [J].
Tsubakimoto, Tatsuya ;
Kobayashi, Kunikazu .
ELECTRONICS AND COMMUNICATIONS IN JAPAN, 2017, 100 (06) :3-10
[42]   Shaping multi-agent systems with gradient reinforcement learning [J].
Buffet, Olivier ;
Dutech, Alain ;
Charpillet, Francois .
AUTONOMOUS AGENTS AND MULTI-AGENT SYSTEMS, 2007, 15 (02) :197-220
[43]   Swarm Reinforcement Learning for traffic signal control based on cooperative multi-agent framework [J].
Tahifa, Mohammed ;
Boumhidi, Jaouad ;
Yahyaouy, Ali .
2015 INTELLIGENT SYSTEMS AND COMPUTER VISION (ISCV), 2015,
[44]   Closely Cooperative Multi-Agent Reinforcement Learning Based on Intention Sharing and Credit Assignment [J].
Fu, Hao ;
You, Mingyu ;
Zhou, Hongjun ;
He, Bin .
IEEE ROBOTICS AND AUTOMATION LETTERS, 2024, 9 (12) :11770-11777
[45]   UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning [J].
Zhang Jiandong ;
Yang Qiming ;
Shi Guoqing ;
Lu Yi ;
Wu Yong .
JOURNAL OF SYSTEMS ENGINEERING AND ELECTRONICS, 2021, 32 (06) :1421-1438
[46]   Learning Cooperative Behaviours in Adversarial Multi-agent Systems [J].
Wang, Ni ;
Das, Gautham P. ;
Millard, Alan G. .
TOWARDS AUTONOMOUS ROBOTIC SYSTEMS, TAROS 2022, 2022, 13546 :179-189
[47]   LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning [J].
Yang, Mingyu ;
Zhao, Jian ;
Hu, Xunhan ;
Zhou, Wengang ;
Zhu, Jiangcheng ;
Li, Houqiang .
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 35 (NEURIPS 2022), 2022,
[48]   Negotiation agent based on Deep reinforcement learning for multi-agent cooperative distributed predictive control. [J].
Aponte-Rengifo, O. ;
Francisco, M. ;
Vega, P. .
IFAC PAPERSONLINE, 2023, 56 (02) :1496-1501
[49]   Assured Deep Multi-Agent Reinforcement Learning for Safe Robotic Systems [J].
Riley, Joshua ;
Calinescu, Radu ;
Paterson, Colin ;
Kudenko, Daniel ;
Banks, Alec .
AGENTS AND ARTIFICIAL INTELLIGENCE, ICAART 2021, 2022, 13251 :158-180
[50]   Schedule-Based Cooperative Multi-agent Reinforcement Learning for Multi-channel Communication in Wireless Sensor Networks [J].
Sahraoui, Mohamed ;
Bilami, Azeddine ;
Taleb-Ahmed, Abdelmalik .
WIRELESS PERSONAL COMMUNICATIONS, 2022, 122 (04) :3445-3465