Composing Synergistic Macro Actions for Reinforcement Learning Agents

被引：0

作者：

Chen, Yu-Ming ^{[1
]}

Chang, Kaun-Yu ^{[2
]}

Liu, Chien ^{[3
]}

Hsiao, Tsu-Ching ^{[4
]}

Hong, Zhang-Wei ^{[5
]}

Lee, Chun-Yi ^{[4
]}

机构：

[1] Taiwan Semicond Mfg Co TSMC, Hsinchu 300, Taiwan

[2] Avery Design Syst Inc, Taipei 100, Taiwan

[3] Univ Rostock, Fac Comp Sci & Elect Engn, D-18051 Rostock, Germany

[4] Natl Tsing Hua Univ, Dept Comp Sci, Hsinchu 300, Taiwan

[5] MIT, Cambridge, MA 02139 USA

来源：

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS | 2024年 / 35卷 / 05期

关键词：

Task analysis; Reinforcement learning; Markov processes; Learning systems; Behavioral sciences; Artificial neural networks; Planning; Macro actions; Markov decision process (MDP); neural architecture search (NAS); reinforcement learning (RL); synergism; NEURAL-NETWORKS;

D O I：

10.1109/TNNLS.2022.3213606

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Macro actions have been demonstrated to be beneficial for the learning processes of an agent and have encouraged a variety of techniques to be developed for constructing more effective ones. However, previous techniques usually do not further consider combining macro actions to form a synergistic macro action ensemble, in which synergism exhibits when the constituent macro actions are favorable to be jointly used by an agent during evaluation. Such a synergistic macro action ensemble may potentially allow an agent to perform even better than the individual macro actions within it. Motivated by the recent advances of neural architecture search (NAS), in this brief, we formulate the construction of a synergistic macro action ensemble as a Markov decision process (MDP) and evaluate the constructed macro action ensemble as a whole. Such a problem formulation enables synergism to be taken into account by the proposed evaluation procedure. Our experimental results demonstrate that the proposed framework is able to discover the synergistic macro action ensembles. Furthermore, we also highlight the benefits of these macro action ensembles through a set of analytical cases.

引用

页码：7251 / 7258

页数：8

共 44 条

[21]

Lakshminarayanan B, 2017, Arxiv, DOI arXiv:1612.01474

[22]

Lee KM, 2018, Arxiv, DOI arXiv:1807.03888

[23] Universal Adversarial Perturbation via Prior Driven Uncertainty Approximation [J].