Comparative Analysis of Deep Learning Models for Breast Cancer Classification on Multimodal Data

Rising breast cancer incidence and mortality represent significant global challenges for women. Deep learning has demonstrated superior diagnostic performance in breast cancer classification compared to human experts. However, most deep learning methods have relied on unimodal features, potentially limiting the performance of diagnostic models. Additionally, most studies conducted so far have used a single view of digital mammograms, which significantly reduces model performance due to limited overall perspective and generalizability. To address these limitations, we collected a multiview multimodal dataset, including digital mammograms four views two craniocaudal (CC), two mediolateral oblique (MLO) one for each breast, and textual data extracted from radiological reports. We propose a multimodal deep learning architecture for breast cancer classification, utilizing images (digital mammograms) and textual data (radiological reports) from our new in-house dataset. In addition, various augmentation techniques are applied to both imaging and textual data to enhance the training data size. In our investigation, we explored the performance of six state-of-the-art (SOTA) deep learning architectures: VGG16, VGG19, ResNet34, MobileNetV3, EfficientNetB7, and a vision transformer (ViT) as an imaging feature extractors. For textual feature extraction, we employed an artificial neural network (ANN). Afterwards, features were fused using an early fusion and late fusion strategy. The fused imaging and textual features were then inputted into an ANN classifier for breast cancer classification. We evaluated various feature extractors and an ANN classifier combinations, finding that VGG19 in association with ANN achieved the highest accuracy at 0.951. In terms of precision, again VGG19 and ANN combination surpassed other SOTA CNN and attention-based architectures, achieving a score of 0.95. The best sensitivity score of 0.893 was recorded by VGG16+ANN, followed by VGG19+ANN with 0.884. The highest F1 score of 0.922 was achieved by VGG19+ANN. VGG16+ANN achieved the best area under the curve (AUC) score of 0.929, closely followed by VGG19+ANN with a score of 0.915.