MuMoSNet: 3D MRI-based Brain Tumor Segmentation via Multi-modal and Multi-scale Feature Fusion

MRI images contain multi-modal information, introducing complexity to brain tumor segmentation. Recent studies have incorporated the Transformer model, given its exceptional capability to model long-range dependence, into convolutional neural networks (CNNs) to address limited receptive fields. However, such a hybrid strategy often neglects the inherent multi-modal characteristics of MRI images and lacks the capacity to capture modality-specific features. In this paper, we propose a multi-modal and multi-scale feature fusion network (MuMoSNet) for brain tumor segmentation from 3D MRI images. Specifically, our MuMoSNet introduces a parallel ME-Transformer encoder alongside the CNN-based encoder in 3D U-Net to separately extract modality-specific features. Besides, we devise a multi-feature fusion (MuFF) module to learn affinity relationships between cross-modality shared features and modality-specific features, maximizing the exploration of multi-modal information. Extensive experiments on both BraTS21 and BraTS20 datasets suggest that our MuMoSNet outperforms current representative methods for brain tumor segmentation.