Multi-layer segmentation framework for cell nuclei using improved GVF Snake model, Watershed, and ellipse fitting

Screening of diseases including cervical cancer, breast cancer and colorectal cancer using cell images from pap smears have been widely applied in recent years. Accurate segmentation of cells including nuclei are important steps of the diagnosis and analysis. Separation of touching nuclei in microscopy images has become a challenge in biological and medical fields. Aiming at the specific screening scenes with limited dataset and labels, a novel multi-layer segmentation framework is proposed, by which the nuclei are segmented efficiently and accurately. First the Watershed and the improved GVF (gradient vector flow) Snake model (optimized by the rectified gradient) are combined for rough segmentation, which preliminarily separates the nuclei from background. Then the outputs from rough segmentation are used for the initialization of refined segmentation, overlapped nuclei is addressed by the convex hull detection, concave point detection and ellipse fitting. Two criteria for ellipse verification are presented to ensure the uniqueness of the fitted border. Better accuracy, robustness and lighter structure of nuclei splitting are gained. Two private (BJTU & BIT) and two publicly available dataset (U20S & NIH3T3) are used for the performance evaluation. Experimental results showed the nuclei are segmented by the proposed framework with effective results (sensitivity: 0.812, precision: 0.913, false positive: 1.8, false negative: 1.7). Proposed method outperforms deep learning models with lighter weight and less computation complexity, it can effectively and automatically segment nuclei or cells from smear images, especially for scenes without sufficient dataset and labels.