Segmentation of endoscopy images of anterior nasal cavity using deep learning

Nasal obstruction (NO), which affects one-third of the adult population, is characterized by a blockage in the nasal cavity. Rhinologists commonly employ nasal endoscopy (NE) in the differential diagnosis of NO, along with a focused history and other examinations such as skin prick tests and CT scans. This study aims to establish NE as a reliable standalone diagnostic tool, eliminating the necessity for CT scans and skin prick tests in the diagnosis of NO. However, currently, there is a lack of objective methods to quantify the severity of NO. To address this problem, we used deep learning to identify the anatomical structures of the anterior nasal cavity, which will then be graded by an objective grading system. In this paper, we evaluated the performance of various deep learning methods (DeepLabv3+, MaskFormer, and Mask2Former) with different pre-trained backbones (ResNet-101 - CNN-based, and Swin-Tiny - transformer-based), for semantic segmentation of the anterior nasal cavity. Sixty-two participants were examined with NE before and after using a nasal decongestant. For model training and validation, 608 images from 46 participants were utilized, and 171 images from 16 participants were reserved for testing. The fine-tuned Mask2Former with low-light image enhancement achieved a mean intersection-over-union of 81.7% and 61.2% on the validation and testing sets, respectively. These findings represent the first successful semantic segmentation of key anatomical structures within the anterior nasal cavity. These segmented structures will serve as the basis for classifying the severity of NO and diagnosing NO conditions, enabling AI-based consultations in primary care settings such as general practices and remote locations, where access to ENT expertise may be limited.