Factorized visual representations in the primate visual system and deep neural networks

被引:0
作者
Lindsey, Jack W. [1 ,2 ]
Issa, Elias B. [1 ,2 ]
机构
[1] Columbia Univ, Zuckerman Mind Brain Behav Inst, New York, NY 10027 USA
[2] Columbia Univ, Dept Neurosci, New York, NY 10027 USA
关键词
visual cortex; deep neural networks; neurophysiology; visual scenes; object recognition; fMRI; OBJECT; SELECTIVITY; INCREASES; FRAMEWORK; GEOMETRY;
D O I
10.7554/eLife.91685; 10.7554/eLife.91685.3.sa1; 10.7554/eLife.91685.3.sa2; 10.7554/eLife.91685.3.sa3
中图分类号
Q [生物科学];
学科分类号
07 ; 0710 ; 09 ;
摘要
Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether ('invariance'), represented in non-interfering subspaces of population activity ('factorization') or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters - lighting, background, camera viewpoint, and object pose - in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI, and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not as consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is often preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.
引用
收藏
页数:20
相关论文
共 56 条
[1]  
Alleman M, 2024, Arxiv, DOI [arXiv:2401.13558, 10.48550/arXiv.2401.13558, DOI 10.48550/ARXIV.2401.13558]
[2]   The Geometry of Abstraction in the Hippocampus and Prefrontal Cortex [J].
Bernardi, Silvia ;
Benna, Marcus K. ;
Rigotti, Mattia ;
Munuera, Jerome ;
Fusi, Stefano ;
Salzman, C. Daniel .
CELL, 2020, 183 (04) :954-+
[3]   Tuned geometries of hippocampal representations meet the computational demands of social memory [J].
Boyle, Lara M. ;
Posani, Lorenzo ;
Irfan, Sarah ;
Siegelbaum, Steven A. ;
Fusi, Stefano .
NEURON, 2024, 112 (08) :1358-1371.e9
[4]   Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition [J].
Cadieu, Charles F. ;
Hong, Ha ;
Yamins, Daniel L. K. ;
Pinto, Nicolas ;
Ardila, Diego ;
Solomon, Ethan A. ;
Majaj, Najib J. ;
DiCarlo, James J. .
PLOS COMPUTATIONAL BIOLOGY, 2014, 10 (12)
[5]  
Caron M, 2019, Arxiv, DOI arXiv:1807.05520
[6]  
Caron M, 2021, Arxiv, DOI arXiv:2006.09882
[7]  
Chen T, 2020, Arxiv, DOI [arXiv:2002.05709, 10.48550/arXiv.2002.05709]
[8]   Classification and Geometry of General Perceptual Manifolds [J].
Chung, SueYeon ;
Lee, Daniel D. ;
Sompolinsky, Haim .
PHYSICAL REVIEW X, 2018, 8 (03)
[9]  
Conwell C, 2022, bioRxiv, DOI [10.1101/2022.03.28.485868, 10.1101/2022.03.28.485868, DOI 10.1101/2022.03.28.485868, 10.1101/2022.03.28.485868v1]
[10]   Untangling invariant object recognition [J].
DiCarlo, James J. ;
Cox, David D. .
TRENDS IN COGNITIVE SCIENCES, 2007, 11 (08) :333-341