Audiovisual integration in the human perception of materials

被引:38
作者
Fujisaki, Waka [1 ]
Goda, Naokazu [2 ]
Motoyoshi, Isamu [3 ]
Komatsu, Hidehiko [2 ]
Nishida, Shin'ya [3 ]
机构
[1] Natl Inst Adv Ind Sci & Technol, Human Technol Res Inst, Tsukuba, Ibaraki, Japan
[2] Natl Inst Physiol Sci, Div Sensory & Cognit Informat, Okazaki, Aichi 444, Japan
[3] NTT Corp, NTT Commun Sci Labs, Atsugi, Kanagawa, Japan
关键词
material perception; audio-visual integration; Bayesian integration; surface texture; impact sound; VISUAL-PERCEPTION; SURFACE; REPRESENTATION; TEXTURE; COLOR; FMRI; FORM; INFORMATION; GLOSS;
D O I
10.1167/14.4.12
中图分类号
R77 [眼科学];
学科分类号
100212 ;
摘要
Interest in the perception of the material of objects has been growing. While material perception is a critical ability for animals to properly regulate behavioral interactions with surrounding objects (e.g., eating), little is known about its underlying processing. Vision and audition provide useful information for material perception; using only its visual appearance or impact sound, we can infer what an object is made from. However, what material is perceived when the visual appearance of one material is combined with the impact sound of another, and what are the rules that govern cross-modal integration of material information? We addressed these questions by asking 16 human participants to rate how likely it was that audiovisual stimuli (48 combinations of visual appearances of six materials and impact sounds of eight materials) along with visual-only stimuli and auditory-only stimuli fell into each of 13 material categories. The results indicated strong interactions between audiovisual material perceptions; for example, the appearance of glass paired with a pepper sound is perceived as transparent plastic. Rating materialcategory likelihoods follow amultiplicative integration rule in that the categories judged to be likely are consistent with both visual and auditory stimuli. On the other hand, rating-material properties, such as roughness and hardness, follow a weighted average rule. Despite a difference in their integration calculations, both rules can be interpreted as optimal Bayesian integration of independent audiovisual estimations for the two types of material judgment, respectively.
引用
收藏
页数:20
相关论文
共 50 条
[41]   Audiovisual integration in low vision individuals [J].
Targher, Stefano ;
Occelli, Valeria ;
Zampini, Massimiliano .
NEUROPSYCHOLOGIA, 2012, 50 (05) :576-582
[42]   Audiovisual integration in familiar person recognition [J].
Schweinberger, Stefan R. ;
Robertson, David M. C. .
VISUAL COGNITION, 2017, 25 (4-6) :589-610
[43]   Audiovisual Integration of Speech in a Bistable Illusion [J].
Munhall, K. G. ;
ten Hove, M. W. ;
Brammer, M. ;
Pare, M. .
CURRENT BIOLOGY, 2009, 19 (09) :735-739
[44]   Shifts in Maximum Audiovisual Integration with Age [J].
Jansen, Samantha D. ;
Keebler, Joseph R. ;
Chaparro, Alex .
MULTISENSORY RESEARCH, 2018, 31 (3-4) :191-212
[45]   Neural Correlates of Interindividual Differences in Children's Audiovisual Speech Perception [J].
Nath, Audrey R. ;
Fava, Eswen E. ;
Beauchamp, Michael S. .
JOURNAL OF NEUROSCIENCE, 2011, 31 (39) :13963-13971
[46]   Audiovisual n-Back Training Alters the Neural Processes of Working Memory and Audiovisual Integration: Evidence of Changes in ERPs [J].
Guo, Ao ;
Yang, Weiping ;
Yang, Xiangfu ;
Lin, Jinfei ;
Li, Zimo ;
Ren, Yanna ;
Yang, Jiajia ;
Wu, Jinglong .
BRAIN SCIENCES, 2023, 13 (07)
[47]   Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex [J].
Salmi, Juha ;
Koistinen, Olli-Pekka ;
Glerean, Enrico ;
Jylanki, Pasi ;
Vehtari, Aki ;
Jaaskelainen, Iiro P. ;
Makela, Sasu ;
Nummenmaa, Lauri ;
Nummi-Kuisma, Katarina ;
Nummi, Ilari ;
Sams, Mikko .
NEUROIMAGE, 2017, 157 :108-117
[48]   Neural Integration of Audiovisual Sensory Inputs in Macaque Amygdala and Adjacent Regions [J].
Shan, Liang ;
Yuan, Liu ;
Zhang, Bo ;
Ma, Jian ;
Xu, Xiao ;
Gu, Fei ;
Jiang, Yi ;
Dai, Ji .
NEUROSCIENCE BULLETIN, 2023, 39 (12) :1749-1761
[49]   Lipreading and Audiovisual Speech Recognition Across the Adult Lifespan: Implications for Audiovisual Integration [J].
Tye-Murray, Nancy ;
Spehar, Brent ;
Myerson, Joel ;
Hale, Sandra ;
Sommers, Mitchell .
PSYCHOLOGY AND AGING, 2016, 31 (04) :380-389
[50]   Influences of selective adaptation on perception of audiovisual speech [J].
Dias, James W. ;
Cook, Theresa C. ;
Rosenblum, Lawrence D. .
JOURNAL OF PHONETICS, 2016, 56 :75-84