Identifying Critical Tokens for Accurate Predictions in Transformer-Based Medical Imaging Models

被引:0
作者
Kang, Solha [1 ]
Vankerschaver, Joris [1 ,2 ]
Ozbulak, Utku [1 ,3 ]
机构
[1] Ghent Univ Global Campus, Ctr Biosyst & Biotech Data Sci, Incheon, South Korea
[2] Univ Ghent, Dept Appl Math Comp Sci & Stat, Ghent, Belgium
[3] Univ Ghent, Dept Elect & Informat Syst, Ghent, Belgium
来源
MACHINE LEARNING IN MEDICAL IMAGING, PT II, MLMI 2024 | 2025年 / 15242卷
关键词
D O I
10.1007/978-3-031-73290-4_17
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
With the advancements in self-supervised learning (SSL), transformer-based computer vision models have recently demonstrated superior results compared to convolutional neural networks (CNNs) and are poised to dominate the field of artificial intelligence (AI)-based medical imaging in the upcoming years. Nevertheless, similar to CNNs, unveiling the decision-making process of transformer-based models remains a challenge. In this work, we take a step towards demystifying the decision-making process of transformer-based medical imaging models and propose "Token Insight", a novel method that identifies the critical tokens that contribute to the prediction made by the model. Our method relies on the principled approach of token discarding native to transformer-based models, requires no additional module, and can be applied to any transformer model. Using the proposed approach, we quantify the importance of each token based on its contribution to the prediction and enable a more nuanced understanding of the model's decisions. Our experimental results which are showcased on the problem of colonic polyp identification using both supervised and self-supervised pretrained vision transformers indicate that Token Insight contributes to a more transparent and interpretable transformer-based medical imaging model, fostering trust and facilitating broader adoption in clinical settings.
引用
收藏
页码:169 / 179
页数:11
相关论文
共 30 条
[21]   ImageNet Large Scale Visual Recognition Challenge [J].
Russakovsky, Olga ;
Deng, Jia ;
Su, Hao ;
Krause, Jonathan ;
Satheesh, Sanjeev ;
Ma, Sean ;
Huang, Zhiheng ;
Karpathy, Andrej ;
Khosla, Aditya ;
Bernstein, Michael ;
Berg, Alexander C. ;
Fei-Fei, Li .
INTERNATIONAL JOURNAL OF COMPUTER VISION, 2015, 115 (03) :211-252
[22]  
Serrano S, 2019, Arxiv, DOI [arXiv:1906.03731, DOI 10.48550/ARXIV.1906.03731]
[23]   Transformers in medical imaging: A survey [J].
Shamshad, Fahad ;
Khan, Salman ;
Zamir, Syed Waqas ;
Khan, Muhammad Haris ;
Hayat, Munawar ;
Khan, Fahad Shahbaz ;
Fu, Huazhu .
MEDICAL IMAGE ANALYSIS, 2023, 88
[24]  
Simonyan K, 2015, Arxiv, DOI [arXiv:1409.1556, DOI 10.48550/ARXIV.1409.1556]
[25]  
Singhal K., 2023, PREPRINT, DOI [arXiv:2305.09617, 10.48550/arXiv.2305.09617, DOI 10.48550/ARXIV.2305.09617, DOI 10.1038/S41591-024-03423-7]
[26]   Right for the Wrong Reason: Can Interpretable ML Techniques Detect Spurious Correlations? [J].
Sun, Susu ;
Koch, Lisa M. ;
Baumgartner, Christian F. .
MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION, MICCAI 2023, PT II, 2023, 14221 :425-434
[27]  
Vaswani A, 2017, ADV NEUR IN, V30
[28]   The effect of yoga on sleep quality and insomnia in women with sleep problems: a systematic review and meta-analysis [J].
Wang, Wei-Li ;
Chen, Kuang-Huei ;
Pan, Ying-Chieh ;
Yang, Szu-Nian ;
Chan, Yuan-Yu .
BMC PSYCHIATRY, 2020, 20 (01)
[29]   Visualizing and Understanding Convolutional Networks [J].
Zeiler, Matthew D. ;
Fergus, Rob .
COMPUTER VISION - ECCV 2014, PT I, 2014, 8689 :818-833
[30]   Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer [J].
Zeng, Wang ;
Jin, Sheng ;
Liu, Wentao ;
Qian, Chen ;
Luo, Ping ;
Ouyang, Wanli ;
Wang, Xiaogang .
2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2022, :11091-11101