Modularity in Nervous Systems-a Key to Efficient Adaptivity for Deep Reinforcement Learning

被引:2
作者
Schilling, Malte [1 ]
Hammer, Barbara [2 ]
Ohl, Frank W. [3 ,4 ]
Ritter, Helge J. [5 ]
Wiskott, Laurenz [6 ]
机构
[1] Univ Munster, Autonomous Intelligent Syst Grp, Einsteinstr 62, D-48149 Munster, Germany
[2] Bielefeld Univ, Machine Learning Grp, Inspirat 1, D-33619 Bielefeld, Germany
[3] Leibniz Inst Neurobiol, Dept Syst Physiol Learning, Brenneckestr 6, D-39118 Magdeburg, Germany
[4] Otto Von Guericke Univ, Inst Biol, Magdeburg, Germany
[5] Bielefeld Univ, Neuroinformat Grp, Inspirat 1, D-33619 Bielefeld, Germany
[6] Ruhr Univ Bochum, Inst Neural Computat, Bochum, Germany
关键词
Reinforcement learning; Motor control; Hierarchical organization; Modularization; Neural networks; CORTICAL ACTIVITY; MOTOR PRIMITIVES; INTERNAL-MODELS; NEURAL BASIS; BEHAVIOR; NEUROSCIENCE; COGNITION;
D O I
10.1007/s12559-022-10080-w
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Modularity as observed in biological systems has proven valuable for guiding classical motor theories towards good answers about action selection and execution. New challenges arise when we turn to learning: Trying to scale current computational models, such as deep reinforcement learning (DRL), to action spaces, input dimensions, and time horizons seen in biological systems still faces severe obstacles unless vast amounts of training data are available. This leads to the question: does biological modularity also hold an important key for better answers to obtain efficient adaptivity for deep reinforcement learning? We review biological experimental work on modularity in biological motor control and link this with current examples of (deep) RL approaches. Analyzing outcomes of simulation studies, we show that these approaches benefit from forms of modularization as found in biological systems. We identify three different strands of modularity exhibited in biological control systems. Two of them-modularity in state (i) and in action (ii) spaces-appear as a consequence of local interconnectivity (as in reflexes) and are often modulated by higher levels in a control hierarchy. A third strand arises from chunking of action elements along a (iii) temporal dimension. Usually interacting in an overarching spatio-temporal hierarchy of the overall system, the three strands offer major "factors" decomposing the entire modularity structure. We conclude that modularity with its above strands can provide an effective prior for DRL approaches to speed up learning considerably and making learned controllers more robust and adaptive.
引用
收藏
页码:2358 / 2373
页数:16
相关论文
共 106 条
[1]   Neural reuse: A fundamental organizational principle of the brain [J].
Anderson, Michael L. .
BEHAVIORAL AND BRAIN SCIENCES, 2010, 33 (04) :245-+
[2]  
[Anonymous], 2009, P 26 ANN INT C MACHI, DOI [DOI 10.1145/1553374.1553380, 10.1145/1553374.155338]
[3]   Connecting neuronal circuits for movement Dedicated neuronal circuits mediate execution, choice, and coordination of body action [J].
Arber, Silvia ;
Costa, Rui M. .
SCIENCE, 2018, 360 (6396) :1403-1404
[4]  
Aubret A, ARXIV
[5]   Shape Similarity, Better than Semantic Membership, Accounts for the Structure of Visual Object Representations in a Population of Monkey Inferotemporal Neurons [J].
Baldassi, Carlo ;
Alemi-Neissi, Alireza ;
Pagan, Marino ;
DiCarlo, James J. ;
Zecchina, Riccardo ;
Zoccolan, Davide .
PLOS COMPUTATIONAL BIOLOGY, 2013, 9 (08)
[6]   Grounded cognition [J].
Barsalou, Lawrence W. .
ANNUAL REVIEW OF PSYCHOLOGY, 2008, 59 :617-645
[7]   The ''independent components'' of natural scenes are edge filters [J].
Bell, AJ ;
Sejnowski, TJ .
VISION RESEARCH, 1997, 37 (23) :3327-3338
[8]   Six-legged walking in insects: how CPGs, peripheral feedback, and descending signals generate coordinated and adaptive motor rhythms [J].
Bidaye, Salil S. ;
Bockemuehl, Till ;
Bueschges, Ansgar .
JOURNAL OF NEUROPHYSIOLOGY, 2018, 119 (02) :459-475
[9]  
Binder MD, 2009, MOTOR CONTROL HIERAR, P2428
[10]   Hierarchical models of behavior and prefrontal function [J].
Botvinick, Matthew M. .
TRENDS IN COGNITIVE SCIENCES, 2008, 12 (05) :201-208