ResMT: A hybrid CNN-transformer framework for glioma grading with 3D MRI

被引:5
作者
Cui, Honghao [1 ]
Ruan, Zhuoying [2 ]
Xu, Zhijian [1 ]
Luo, Xiao [1 ]
Dai, Jian [1 ]
Geng, Daoying [1 ,2 ]
机构
[1] Fudan Univ, Acad Engn & Technol, Shanghai 200433, Peoples R China
[2] Fudan Univ, Huashan Hosp, Dept Radiol, Shanghai 200040, Peoples R China
关键词
Glioma grading; Deep learning; Hybrid architecture; Attention mechanisms; Transformer; Magnetic resonance imaging; CENTRAL-NERVOUS-SYSTEM; NETWORK;
D O I
10.1016/j.compeleceng.2024.109745
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
Accurate grading of gliomas is crucial for treatment strategies and prognosis. While convolutional neural networks (CNNs) have proven effective in classifying medical images, they struggle with capturing long-range dependencies among pixels. Transformer-based networks can address this issue, but CNN-based methods often perform better when trained on small datasets. Additionally, tumor segmentation is essential for classification models, but training an additional segmentation model significantly increases workload. To address these challenges, we propose ResMT, which combines CNN and transformer architectures for glioma grading, extracting both local and global features efficiently. Specifically, we designed a spatial residual module (SRM) where a 3D CNN captures glioma's volumetric complexity, and Swin UNETR, a pre-trained segmentation model, enhances the network without extra training. Our model also includes a multi-plane channel and spatial attention module (MCSA) to refine the analysis by focusing on critical features across multiple planes (axial, coronal, and sagittal). Transformer blocks establish long-range relationships among planes and slices. We evaluated ResMT on the BraTs19 dataset, comparing it with baselines and state-of-the-art models. Results demonstrate that ResMT achieves the highest prediction performance with an AUC of 0.9953, highlighting hybrid CNN-transformer models' potential for 3D MRI classification.
引用
收藏
页数:17
相关论文
共 50 条
[41]   A CNN-Transformer Hybrid Approach for Crop Classification Using Multitemporal Multisensor Images [J].
Li, Zhengtao ;
Chen, Guokun ;
Zhang, Tianxu .
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, 2020, 13 :847-858
[42]   HYBRID CNN-TRANSFORMER FOR MARINE AQUACULTURE SEMANTIC SEGMENTATION BASED ON POLSAR IMAGES [J].
Liu, Keyuan ;
Zheng, Danchen ;
Fan, Jianchao .
2024 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS 2024), 2024, :8255-8258
[43]   A Lightweight 3D Distillation Volumetric Transformer for 3D MRI Super-Resolution [J].
Zhao, Jianwei ;
Hong, Tao ;
Qi, Hao ;
Zhou, Zhenghua ;
Wang, Hai .
IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, 2025, 29 (07) :5083-5094
[44]   A CNN-TRANSFORMER HYBRID FEATURE DESCRIPTOR FOR OPTICAL-SAR IMAGE REGISTRATION [J].
Lin, Mingxin ;
Liu, Binyuan ;
Liu, Yijun ;
Wang, Qingsong .
IGARSS 2023 - 2023 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM, 2023, :6069-6072
[45]   TDGU-Net: A Hybrid CNN-Transformer Model for Intracranial Aneurysm Segmentation [J].
Lin, Xiaoqing ;
Wang, Chen ;
Chen, Zhengkui ;
Pan, Jianwei ;
Tong, Jijun .
INTERNATIONAL JOURNAL OF IMAGING SYSTEMS AND TECHNOLOGY, 2025, 35 (04)
[46]   Enhancing structural illumination microscopy with hybrid CNN-transformer and dynamic frequency loss [J].
Liu, Tao ;
Niu, Jingwei ;
Liu, Jiahao ;
Li, Dong ;
Tan, Shan .
OPTICS AND LASERS IN ENGINEERING, 2024, 173
[47]   An Efficient Hybrid CNN-Transformer Approach for Remote Sensing Super-Resolution [J].
Zhang, Wenjian ;
Tan, Zheng ;
Lv, Qunbo ;
Li, Jiaao ;
Zhu, Baoyu ;
Liu, Yangyang .
REMOTE SENSING, 2024, 16 (05)
[48]   DACTransNet: A Hybrid CNN-Transformer Network for Histopathological Image Classification of Pancreatic Cancer [J].
Kou, Yongqing ;
Xia, Cong ;
Jiao, Yiping ;
Zhang, Daoqiang ;
Ge, Rongjun .
ARTIFICIAL INTELLIGENCE, CICAI 2023, PT II, 2024, 14474 :422-434
[49]   HAU-Net: Hybrid CNN-transformer for breast ultrasound image segmentation [J].
Zhang, Huaikun ;
Lian, Jing ;
Yi, Zetong ;
Wu, Ruichao ;
Lu, Xiangyu ;
Ma, Pei ;
Ma, Yide .
BIOMEDICAL SIGNAL PROCESSING AND CONTROL, 2024, 87
[50]   Laser Stripe Segmentation of Weld Seam Based on CNN-Transformer Hybrid Networks [J].
Wang, Ying ;
Gao, Sheng ;
Dai, Zhe .
CHINESE JOURNAL OF LASERS-ZHONGGUO JIGUANG, 2024, 51 (24)