The neural architecture of theory-based reinforcement learning

被引:7
|
作者
Tomov, Momchil S. [1 ,2 ,4 ,5 ]
Tsividis, Pedro A. [3 ,4 ]
Pouncy, Thomas [1 ,2 ]
Tenenbaum, Joshua B. [3 ,4 ]
Gershman, Samuel J. [1 ,2 ,4 ]
机构
[1] Harvard Univ, Dept Psychol, Cambridge, MA 02138 USA
[2] Harvard Univ, Ctr Brain Sci, Cambridge, MA 02138 USA
[3] MIT, Dept Brain & Cognit Sci, Cambridge, MA 02139 USA
[4] MIT, Ctr Brains Minds & Machines, Cambridge, MA 02139 USA
[5] Mot AD Inc, Boston, MA 02210 USA
关键词
ORBITOFRONTAL CORTEX; CAUSAL INFERENCE; COGNITIVE MAPS; MODEL; PREDICTION; BRAIN; GO; REPRESENTATIONS; KNOWLEDGE; HUMANS;
D O I
10.1016/j.neuron.2023.01.023
中图分类号
Q189 [神经科学];
学科分类号
071006 ;
摘要
Humans learn internal models of the world that support planning and generalization in complex environ-ments. Yet it remains unclear how such internal models are represented and learned in the brain. We approach this question using theory-based reinforcement learning, a strong form of model-based reinforce-ment learning in which the model is a kind of intuitive theory. We analyzed fMRI data from human participants learning to play Atari-style games. We found evidence of theory representations in prefrontal cortex and of theory updating in prefrontal cortex, occipital cortex, and fusiform gyrus. Theory updates coincided with transient strengthening of theory representations. Effective connectivity during theory updating suggests that information flows from prefrontal theory-coding regions to posterior theory-updating regions. Together, our results are consistent with a neural architecture in which top-down theory representations originating in prefrontal regions shape sensory predictions in visual areas, where factored theory prediction errors are computed and trigger bottom-up updates of the theory.
引用
收藏
页码:1331 / +
页数:23
相关论文
共 50 条
  • [1] Inductive biases in theory-based reinforcement learning
    Pouncy, Thomas
    Gershman, Samuel J.
    COGNITIVE PSYCHOLOGY, 2022, 138
  • [2] Learning to Troubleshoot: A New Theory-Based Design Architecture
    David H. Jonassen
    Woei Hung
    Educational Psychology Review, 2006, 18 : 77 - 114
  • [3] Learning to troubleshoot: A new theory-based design architecture
    Jonassen, David H.
    Hung, Woei
    EDUCATIONAL PSYCHOLOGY REVIEW, 2006, 18 (01) : 77 - 114
  • [4] Renewal Monte Carlo: Renewal Theory-Based Reinforcement Learning
    Subramanian, Jayakumar
    Mahajan, Aditya
    IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 2020, 65 (08) : 3663 - 3670
  • [5] A Theory-Based Explainable Deep Learning Architecture for Music Emotion
    Fong, Hortense
    Kumar, Vineet
    Sudhir, K.
    MARKETING SCIENCE, 2025, 44 (01)
  • [6] Spectral graph theory-based virtual network embedding for vehicular fog computing: A deep reinforcement learning architecture
    Chen, Ning
    Zhang, Peiying
    Kumar, Neeraj
    Hsu, Ching-Hsien
    Abualigah, Laith
    Zhu, Hailong
    KNOWLEDGE-BASED SYSTEMS, 2022, 257
  • [7] Reinforcement Learning based Neural Architecture Search for Audio Tagging
    Liu, Haiyang
    Zhang, Cheng
    2020 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), 2020,
  • [8] Scalable reinforcement learning-based neural architecture search
    Amber Cassimon
    Siegfried Mercelis
    Kevin Mets
    Neural Computing and Applications, 2025, 37 (1) : 231 - 261
  • [9] Optimizing the Neural Architecture of Reinforcement Learning Agents
    Mazyavkina, N.
    Moustafa, S.
    Trofimov, I
    Burnaev, E.
    INTELLIGENT COMPUTING, VOL 2, 2021, 284 : 591 - 606
  • [10] A Neural Architecture to Address Reinforcement Learning Problems
    de Arruda, Rodrigo L. S.
    Von Zuben, Fernando J.
    2011 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), 2011, : 2930 - 2935