Automatic Mitosis and Nuclear Atypia Detection for Breast Cancer Grading in Histopathological Images using Hybrid Machine Learning Technique

Invasive breast cancer is a complex global health issue and the leading cause of women's mortality. Multiclassification in breast cancer, especially with high-resolution images, presents unique challenges. Clinical diagnosis relies on the cancer's pathological stage, requiring precise segmentation and adjustments. Complex structural changes during slide preparation and inconsistent image magnifications further complicate classification. To address these challenges, we propose a hybrid machine learning framework for accurate breast cancer detection and grading using large-scale pathological images. Our approach includes an improved Non-restricted Boltzmann Deep Belief Neural Network for nuclei segmentation, followed by feature extraction and novel feature selection using the Giraffe Kicking Optimization algorithm to mitigate overfitting. We implement an Optimal Kernel layer-based Support Vector Machine classifier to identify mitotic cells and nuclear atypia, using the Nottingham Grading System. Validation on the MITOSIS-ATYPIA-14 database demonstrates the framework's effectiveness, with performance metrics including accuracy, precision, recall, specificity, and F-measure. This approach addresses the complexities of breast cancer classification and grading in a streamlined manner, enhancing diagnostic accuracy and prognosis prediction. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.