A survey of deep learning methods on cell instance segmentation

Cell segmentation is a key topic in medical image analysis with a wide range of applications in the study of diagnosis and prognosis of pathology and cytology. Along with the recent development of generative adversarial networks and transformers, there has been a substantial amount of work aimed at developing cell segmentation approaches using deep learning (DL) models. Inspired by this transition, in this survey, we provide a comprehensive review of the current situation and future technology development in cell instance segmentation by systematically reviewing 198 research papers, covering a broad spectrum of models for instance-level cell segmentation from 2020 to 2024, including convolutional networks, encoder–decoder architectures, recurrent networks, transformers and generative adversarial models. We have examined the loss functions, training strategies, evaluation methods, widely used datasets and quantitative performance of individual methods. A comprehensive summary of the selected seminal works on DL-based cell segmentation with microscopic images is further provided to investigate the effectiveness of methods. We have also performed a comparative analysis on two challenging cell instance segmentation datasets with technical challenges, including unclear cell boundaries, clustered or overlapping cells, variations in cell appearance and sparse or missing annotations, utilizing 18 state-of-the-art DL approaches in cell instance segmentation. Finally, we described the strengths and challenges of the cell instance segmentation models with discussions on future research directions in this area.