Crossmodal Language Comprehension-Psycholinguistic Insights and Computational Approaches

被引:3
|
作者
Alacam, Ozge [1 ]
Li, Xingshan [2 ]
Menzel, Wolfgang [1 ]
Staron, Tobias [1 ]
机构
[1] Univ Hamburg, Dept Informat, Nat Language Syst Grp, Hamburg, Germany
[2] Chinese Acad Sci, Inst Psychol, Reading & Visual Cognit Lab, Beijing, Peoples R China
来源
FRONTIERS IN NEUROROBOTICS | 2020年 / 14卷 / 14期
基金
中国国家自然科学基金;
关键词
language comprehension; crossmodality; psycholinguistics; incrementality; prediction; speaker intention; EYE-MOVEMENTS; WORLD KNOWLEDGE; SEARCH; INCREMENTALITY; RECOGNITION; CONSTRAINTS; INFORMATION; INTEGRATION; LISTENERS; DISCOURSE;
D O I
10.3389/fnbot.2020.00002
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn plays a role in driving the focus of attention in the visual environment. However, language and vision are two different representational modalities, which accommodate different aspects and granularities of conceptualizations. To integrate them into a single, coherent system solution is still a challenge, which could profit from inspiration by human crossmodal processing. Based on fundamental psycholinguistic insights into the nature of situated language comprehension, we derive a set of performance characteristics facilitating the robustness of language understanding, such as crossmodal reference resolution, attention guidance, or predictive processing. Artificial systems for language comprehension should meet these characteristics in order to be able to perform in a natural and smooth manner. We discuss how empirical findings on the crossmodal support of language comprehension in humans can be applied in computational solutions for situated language comprehension and how they can help to mitigate the shortcomings of current approaches.
引用
收藏
页数:17
相关论文
共 50 条
  • [21] A computational model of language comprehension unites diverse perspectives
    MacDonald, Maryellen C.
    PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA, 2022, 119 (49)
  • [22] Computational and experimental approaches for chitosan-based nano PECs design: Insights on a deeper comprehension of nanostructure formation
    Boni, Fernanda Isadora
    Ferreira Cury, Beatriz Stringhetti
    Ferreira, Natalia Noronha
    Teixeira, Deiver Alessandro
    Daflon Gremiao, Maria Palmira
    CARBOHYDRATE POLYMERS, 2021, 254
  • [23] PARSING IN CONTEXT - COMPUTATIONAL AND PSYCHOLINGUISTIC APPROACHES TO RESOLVING AMBIGUITY DURING SENTENCE PROCESSING
    GARNHAM, A
    ALTMANN, G
    LANGUAGE AND COGNITIVE PROCESSES, 1995, 10 (3-4): : 377 - 381
  • [24] Computational approaches to language acquisition.
    Pine, JM
    Gobet, FR
    Rowland, CF
    JOURNAL OF CHILD LANGUAGE, 1999, 26 (01) : 203 - 208
  • [25] Computational Approaches to the Study of Language Change
    Baker, Adam
    LANGUAGE AND LINGUISTICS COMPASS, 2008, 2 (02): : 289 - 307
  • [26] Psycholinguistic approaches to instructed second language acquisition: linking theory, findings and practice
    Dong, Xinrui
    Li, Xukui
    INNOVATION IN LANGUAGE LEARNING AND TEACHING, 2024, 18 (05) : 491 - 493
  • [27] Psycholinguistic approaches to instructed second language acquisition: linking theory, findings and practice
    Liu, Ziman
    Chen, Shifa
    Wang, Shaoxin
    PHILOSOPHICAL PSYCHOLOGY, 2024,
  • [28] Towards a Computational Model of Actor-Based Language Comprehension
    Phillip M. Alday
    Matthias Schlesewsky
    Ina Bornkessel-Schlesewsky
    Neuroinformatics, 2014, 12 : 143 - 179
  • [29] Towards a Computational Model of Actor-Based Language Comprehension
    Alday, Phillip M.
    Schlesewsky, Matthias
    Bornkessel-Schlesewsky, Ina
    NEUROINFORMATICS, 2014, 12 (01) : 143 - 179
  • [30] Non-binary language in Spanish? Comprehension of non-binary morphological forms: a psycholinguistic study
    Stetie, Noelia Ayelen
    Zunino, Gabriela Mariel
    GLOSSA-A JOURNAL OF GENERAL LINGUISTICS, 2022, 7 (01):