Beyond dichotomies in reinforcement learning

被引:73
作者
Collins, Anne G. E. [1 ,2 ]
Cockburn, Jeffrey [3 ]
机构
[1] Univ Calif Berkeley, Dept Psychol, 3210 Tolman Hall, Berkeley, CA 94720 USA
[2] Univ Calif Berkeley, Helen Wills Neurosci Inst, Berkeley, CA 94720 USA
[3] CALTECH, Div Humanities & Social Sci, Pasadena, CA 91125 USA
关键词
DOPAMINE NEURONS ENCODE; MODEL-BASED CONTROL; PREFRONTAL CORTEX; WORKING-MEMORY; INDIVIDUAL-DIFFERENCES; PREDICTION ERRORS; DECISION-MAKING; BASAL GANGLIA; SYSTEMS; REWARD;
D O I
10.1038/s41583-020-0355-6
中图分类号
Q189 [神经科学];
学科分类号
071006 ;
摘要
Reinforcement learning (RL) is a framework of particular importance to psychology, neuroscience and machine learning. Interactions between these fields, as promoted through the common hub of RL, has facilitated paradigm shifts that relate multiple levels of analysis in a singular framework (for example, relating dopamine function to a computationally defined RL signal). Recently, more sophisticated RL algorithms have been proposed to better account for human learning, and in particular its oft-documented reliance on two separable systems: a model-based (MB) system and a model-free (MF) system. However, along with many benefits, this dichotomous lens can distort questions, and may contribute to an unnecessarily narrow perspective on learning and decision-making. Here, we outline some of the consequences that come from overconfidently mapping algorithms, such as MB versus MF RL, with putative cognitive processes. We argue that the field is well positioned to move beyond simplistic dichotomies, and we propose a means of refocusing research questions towards the rich and complex components that comprise learning and decision-making. Reinforcement learning has been suggested to come in two flavours: model-free and model-based. In this Perspective, Collins and Cockburn explain why viewing reinforcement learning through this dichotomous lens is not always accurate or helpful, and suggest paths forward.
引用
收藏
页码:576 / 586
页数:11
相关论文
共 158 条
[1]   Computational Psychiatry: towards a mathematically informed understanding of mental illness [J].
Adams, Rick A. ;
Huys, Quentin J. M. ;
Roiser, Jonathan P. .
JOURNAL OF NEUROLOGY NEUROSURGERY AND PSYCHIATRY, 2016, 87 (01) :53-63
[2]   Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task [J].
Akam, Thomas ;
Costa, Rui ;
Dayan, Peter .
PLOS COMPUTATIONAL BIOLOGY, 2015, 11 (12)
[3]   Knowing how much you don't know: a neural organization of uncertainty estimates [J].
Bach, Dominik R. ;
Dolan, Raymond J. .
NATURE REVIEWS NEUROSCIENCE, 2012, 13 (08) :572-586
[4]   Interactionist Neuroscience [J].
Badre, David ;
Frank, Michael J. ;
Moore, Christopher I. .
NEURON, 2015, 88 (05) :855-860
[5]   Mechanisms of Hierarchical Reinforcement Learning in Cortico-Striatal Circuits 2: Evidence from fMRI [J].
Badre, David ;
Frank, Michael J. .
CEREBRAL CORTEX, 2012, 22 (03) :527-536
[6]   Rostrolateral Prefrontal Cortex and Individual Differences in Uncertainty-Driven Exploration [J].
Badre, David ;
Doll, Bradley B. ;
Long, Nicole M. ;
Frank, Michael J. .
NEURON, 2012, 73 (03) :595-607
[7]   Frontal Cortex and the Discovery of Abstract Action Rules [J].
Badre, David ;
Kayser, Andrew S. ;
D'Esposito, Mark .
NEURON, 2010, 66 (02) :315-326
[8]   Hippocampal pattern separation supports reinforcement learning [J].
Ballard, Ian C. ;
Wagner, Anthony D. ;
McClure, Samuel M. .
NATURE COMMUNICATIONS, 2019, 10 (1)
[9]   Goal-directed instrumental action: contingency and incentive learning and their cortical substrates [J].
Balleine, BW ;
Dickinson, A .
NEUROPHARMACOLOGY, 1998, 37 (4-5) :407-419
[10]   Explicit and Implicit Reinforcement Learning Across the Psychosis Spectrum [J].
Barch, Deanna M. ;
Carter, Cameron S. ;
Gold, James M. ;
Johnson, Sheri L. ;
Kring, Ann M. ;
MacDonald, Angus W., III ;
Pizzagalli, Diego A. ;
Ragland, J. Daniel ;
Silverstein, Steven M. ;
Strauss, Milton E. .
JOURNAL OF ABNORMAL PSYCHOLOGY, 2017, 126 (05) :694-711