Segmentation, Detection and Classification of Cell Nuclei on Oral Cytology Samples Stained with Papanicolaou

Although oral cancer is considered a global health issue with 350.000 people diagnosed over a year it can successfully be treated if diagnosed at early stages. Papanicolaou is an unexpensive and non-invasive method, generally applied to detect cervical cancer, but it can also be useful to detect cancer on oral cavities. The manual process of analyzing cells to detect abnormalities is a time-consuming cell analysis and is subject to variations in perceptions from different professionals. This paper compares models for three different deep learning approaches: segmentation, object detection and image classification. Our results show that the binary object detection with Faster R-CNN is the best approach for nuclei detection and localization (0.76 IoU). Since ResNet 34 had a good performance on abnormal nuclei classification (0.88 accuracy, 0.86 F-1 score) we concluded that these two models can be used in combination to make a localization and classification pipeline. This work reinforces that the automated analysis of oral cytology to make a pipeline for nuclei classification and localization using deep learning can help to minimize the subjectivity of the human analysis and also to detect cancer at early stages.