Explainable Multimodal Learning in Remote Sensing: Challenges and Future Directions

被引:0
作者
Guenther, Alexander [1 ]
Najjar, Hiba [1 ,2 ]
Dengel, Andreas [1 ,2 ]
机构
[1] Univ Kaiserslautern Landau, Dept Comp Sci, D-67663 Kaiserslautern, Germany
[2] German Res Ctr Artificial Intelligence DFKI, Dept Comp Sci, D-67663 Kaiserslautern, Germany
关键词
Deep learning (DL); Earth observation; explainability; interpretability; multimodal learning; remote sensing (RS); ARTIFICIAL-INTELLIGENCE;
D O I
10.1109/LGRS.2024.3404596
中图分类号
P3 [地球物理学]; P59 [地球化学];
学科分类号
0708 ; 070902 ;
摘要
Earth observation applications effectively leverage deep learning (DL) models to harness the abundantly available remote sensing (RS) data. In order to use all the different modalities relevant to a specific task, the fusion of these data sources can be achieved using multimodal learning techniques. This is especially helpful when the input dataset contains both images and tabular data or when the temporal and spatial resolutions vary across the modalities of interest. Nevertheless, these fusion techniques typically increase in complexity, as the disparities in the nature of the fused modalities increase. The resulting complex DL models suffer from a lack of explainability and transparency, which is crucial in many sensitive human-related applications. In this letter, we describe how the research community in RS addresses the issue of model explainability in the context of multimodal learning. We additionally review the practices used in other application fields and identify some of the most promising explainability methods tailored for multimodal deep networks to be exploited in RS applications.
引用
收藏
页码:1 / 5
页数:5
相关论文
共 50 条
[41]   Explainable Multimodal Machine Learning for Engagement Analysis by Continuous Performance Test [J].
Rahman, Muhammad Arifur ;
Brown, David J. ;
Shopland, Nicholas ;
Burton, Andrew ;
Mahmud, Mufti .
UNIVERSAL ACCESS IN HUMAN-COMPUTER INTERACTION: USER AND CONTEXT DIVERSITY, UAHCI 2022, PT II, 2022, 13309 :386-399
[42]   A review of secure federated learning: Privacy leakage threats, protection technologies, challenges and future directions [J].
Ge, Lina ;
Li, Haiao ;
Wang, Xiao ;
Wang, Zhe .
NEUROCOMPUTING, 2023, 561
[43]   The Integration of Deep Learning in Radiotherapy: Exploring Challenges, Opportunities, and Future Directions through an Umbrella Review [J].
Lastrucci, Andrea ;
Wandael, Yannick ;
Ricci, Renzo ;
Maccioni, Giovanni ;
Giansanti, Daniele .
DIAGNOSTICS, 2024, 14 (09)
[44]   Deep learning for SDN-enabled campus networks: proposed solutions, challenges and future directions [J].
Chanhemo, Wilson Charles ;
Mohsini, Mustafa H. ;
Mjahidi, Mohamedi M. ;
Rashidi, Florence U. .
INTERNATIONAL JOURNAL OF INTELLIGENT COMPUTING AND CYBERNETICS, 2023, 16 (04) :697-726
[45]   Deep learning-based natural language processing in ophthalmology: applications, challenges and future directions [J].
Yang, Lily Wei Yun ;
Ng, Wei Yan ;
Foo, Li Lian ;
Liu, Yong ;
Yan, Ming ;
Lei, Xiaofeng ;
Zhang, Xiaoman ;
Ting, Daniel Shu Wei .
CURRENT OPINION IN OPHTHALMOLOGY, 2021, 32 (05) :397-405
[46]   Machine Learning for Prediction of the International Roughness Index on Flexible Pavements: A Review, Challenges, and Future Directions [J].
Tamagusko, Tiago ;
Ferreira, Adelino .
INFRASTRUCTURES, 2023, 8 (12)
[47]   Tutorial on Explainable Deep Reinforcement Learning: One framework, three paradigms and many challenges [J].
Vouros, George A. .
PROCEEDINGS OF THE 12TH HELLENIC CONFERENCE ON ARTIFICIAL INTELLIGENCE, SETN 2022, 2022,
[48]   Multimodal Fusion Remote Sensing Image-Audio Retrieval [J].
Yang, Rui ;
Wang, Shuang ;
Sun, Yingzhi ;
Zhang, Huan ;
Liao, Yu ;
Gu, Yu ;
Hou, Biao ;
Jiao, Licheng .
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, 2022, 15 :6220-6235
[49]   Opening the Black Box: A systematic review on explainable artificial intelligence in remote sensing [J].
Hoehl, Adrian ;
Obadic, Ivica ;
Fernandez-Torres, Miguel-Angel ;
Najjar, Hiba ;
Oliveira, Dario Augusto Borges ;
Akata, Zeynep ;
Dengel, Andreas ;
Zhu, Xiao Xiang .
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE, 2024, 12 (04) :261-304
[50]   Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions [J].
Longo, Luca ;
Brcic, Mario ;
Cabitza, Federico ;
Choi, Jaesik ;
Confalonieri, Roberto ;
Del Ser, Javier ;
Guidotti, Riccardo ;
Hayashi, Yoichi ;
Herrera, Francisco ;
Holzinger, Andreas ;
Jiang, Richard ;
Khosravi, Hassan ;
Lecue, Freddy ;
Malgieri, Gianclaudio ;
Paez, Andres ;
Samek, Wojciech ;
Schneider, Johannes ;
Speith, Timo ;
Stumpf, Simone .
INFORMATION FUSION, 2024, 106