Transfer learning based deep CNN for segmentation and detection of mitoses in breast cancer histopathological images

A Transfer Learning based system is proposed for segmentation and detection of mitotic nuclei. To give the classifier a balanced dataset, first a pre-trained convolutional neural network (CNN) is modified for segmentation of the nuclei. Then another Hybrid-CNN (with Weights Transfer and custom layers) is used for classification of mitoses. Abstract Segmentation and detection of mitotic nuclei is a challenging task. To address this problem, a Transfer Learning based fast and accurate system is proposed. To give the classifier a balanced dataset, this work exploits the concept of Transfer Learning by first using a pre-trained convolutional neural network (CNN) for segmentation, and then another Hybrid-CNN (with Weights Transfer and custom layers) for classification of mitoses. First, mitotic nuclei are automatically annotated, based on the ground truth centroids. The segmentation module then segments mitotic nuclei and also produces some false positives. Finally, the detection module is trained on the patches from the segmentation module and performs the final detection. Fine-tuning based Transfer Learning reduced training time, provided good initial weights, and improved the detection rate with F-measure of 0.713 and 76% area under the precision-recall curve for the challenging task of mitosis detection.