Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"

被引:0
作者
Amizadeh, Saeed [1 ]
Palangi, Hamid [2 ]
Polozov, Oleksandr [2 ]
Huang, Yichen [2 ]
Koishida, Kazuhito [1 ]
机构
[1] Microsoft Appl Sci Grp ASG, Redmond, WA 98052 USA
[2] Microsoft Res AI, Redmond, WA USA
来源
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 119 | 2020年 / 119卷
关键词
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. To this end, we introduce a differentiable first-order logic formalism for VQA that explicitly decouples question answering from visual perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models leading to informative insights regarding the participating models as well as the task.
引用
收藏
页数:12
相关论文
共 50 条
[21]   An energy-based model for neuro-symbolic reasoning on knowledge graphs [J].
Doldy, Dominik ;
Garridoy, Josep Soler .
20TH IEEE INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND APPLICATIONS (ICMLA 2021), 2021, :916-921
[22]   Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsal [J].
Marconato, Emanuele ;
Bontempo, Gianpaolo ;
Ficarra, Elisa ;
Calderara, Simone ;
Passerini, Andrea ;
Teso, Stefano .
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 202, 2023, 202
[23]   Coalition Situational Understanding via Explainable Neuro-Symbolic Reasoning and Learning [J].
Preece, Alun ;
Braines, Dave ;
Cerutti, Federico ;
Furby, Jack ;
Hiley, Liam ;
Kaplan, Lance ;
Law, Mark ;
Russo, Alessandra ;
Srivastava, Mani ;
Vilamala, Marc Roig ;
Xing, Tianwei .
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING FOR MULTI-DOMAIN OPERATIONS APPLICATIONS III, 2021, 11746
[24]   Neuro-symbolic procedural semantics for explainable visual dialogue [J].
Verheyen, Lara ;
Botoko Ekila, Jerome ;
Nevens, Jens ;
Van Eecke, Paul ;
Beuls, Katrien .
PLOS ONE, 2025, 20 (05)
[25]   An Interpretable Neuro-Symbolic Reasoning Framework for Task-Oriented Dialogue Generation [J].
Yang, Shiquan ;
Zhang, Rui ;
Erfani, Sarah ;
Lau, Jey Han .
PROCEEDINGS OF THE 60TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), VOL 1: (LONG PAPERS), 2022, :4918-4935
[26]   Weakly Supervised Neuro-Symbolic Module Networks for Numerical Reasoning over Text [J].
Saha, Amrita ;
Joty, Shafiq ;
Hoi, Steven C. H. .
THIRTY-SIXTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE / THIRTY-FOURTH CONFERENCE ON INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE / TWELVETH SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2022, :11238-11247
[27]   UNDERSTANDING COMPLEX DYNAMICS BY VISUAL AND SYMBOLIC REASONING [J].
YIP, KM .
ARTIFICIAL INTELLIGENCE, 1991, 51 (1-3) :179-221
[28]   Neuro-Symbolic Causal Reasoning Meets Signaling Game for Emergent Semantic Communications [J].
Thomas, Christo Kurisummoottil ;
Saad, Walid .
IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, 2024, 23 (05) :4546-4563
[29]   A review of neuro-symbolic AI integrating reasoning and learning for advanced cognitive systems [J].
Nawaz, Uzma ;
Anees-ur-Rahaman, Mufti ;
Saeed, Zubair .
INTELLIGENT SYSTEMS WITH APPLICATIONS, 2025, 26
[30]   Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts [J].
Marconato, Emanuele ;
Teso, Stefano ;
Vergari, Antonio ;
Passerini, Andrea .
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 36 (NEURIPS 2023), 2023,