XAI-RACapsNet: Relevance aware capsule network-based breast cancer detection using mammography images via explainability O-net ROI segmentation

Breast cancer is a malignant condition characterized by the uncontrolled growth of abnormal cells in breast tissues, often forming a lump or mass that can be detected through screening or self-examination. The early detection of breast cancer is crucial, as it enables timely intervention, increases the likelihood of successful treatment, improves outcomes, and enhances survival rates. The existing work poses many limitations in effectively detecting breast cancer using AI on mammogram images, such as potential false positives/negatives, dependence on image quality, and challenges in interpreting subtle abnormalities that may affect the model's accuracy. To improve the model performance and explainability-based detection, in this paper, we propose a novel approach with the basis of a hybrid Explainability and Relevance aware AI based breast cancer detection model named XAI-RACapsNet. In our work, we utilize publicly available mammogram images for further processing. Initially, the bi-level data pre-treatment is performed in terms of noise reduction using Mean Filter (MF) and histogram equalization by Contrast Limited Adaptive Histogram Equalization (CLAHE) with the aim of enhancing the quality of images. The explainable AI-based ROI segmentation approach is then proposed for implementing an effective segmentation method named XAI O-Net, which encompasses a transformer encoder, ResNet, and transformer decoder module. We then employ an Adaptive Feature Extraction Module (AFEM) for extracting appropriate features from the segmented image, including texture, gradient, and geometric. Based on the extracted features, breast cancer classification is performed by designing a Relevance Aware Capsule Network (RACapsNet), and then relevance heat maps are generated. The performance of the proposed XAIRACapsNet methodology is validated by evaluating performance metrics, such as accuracy, sensitivity, specificity, precision, FDR, FNR, FPR, NPV, F1-score and MCC. The experimental results unequivocally highlight the superiority of the proposed XAI-RACapsNet model over existing models.