Automatic Detection and Counting of Lymphocytes from Immunohistochemistry Cancer Images Using Deep Learning

Purpose Cancer is one of the most life-threatening and devastating diseases in the world. The generally recognized standard for cancer staging is the TNM staging system. In addition, a new parameter called the Immunoscore has been developed recently to assess the survival rate of patients. The Immunoscore is based on counts of CD3+ and CD8+ lymphocytes in the tumour core and margin. Counting the number of lymphocytes is a tedious process for pathologists. This paper examines the use of deep learning techniques for automatic detection and counting of lymphocytes from immunohistochemistry images of breast, colon, and prostate cancers. Methods We used an object detector called Faster R-CNN with four feature extractors: Resnet-50, VGG-16, Inception-V2, and Resnet-101 for automatic detection and counting of lymphocytes. A total of 11,136 lymphocytes were annotated after performing data augmentation on 1228 images. The test images are separated into three regions of interest (ROI): scattered lymphocytes, groups of lymphocytes, and artefacts. In each ROI, the performance of the object detector was checked by evaluation metrics. Results On comparing the F1-score for all three ROIs, we found that Resnet-101 provides better performance than the other feature extractors when using Faster R-CNN. The mean error in lymphocyte count for all ROIs appears to be insignificant. The detection time for a single image is less for VGG-16 than for other feature extractors. Conclusion This study presents a fine-tuned Faster R-CNN object detector for automatic detection and counting of lymphocytes in three different cancer tissues for scoring lymphocytes. Our results suggest that the Faster R-CNN method is efficient and yields good results. Thus, the proposed method can assist pathologists in providing a cancer prognosis.