Inter-rater reliability between experienced and inexperienced otolaryngologists using Koo's drug-induced sleep endoscopy classification system

Purpose An ideal, drug-induced sleep endoscopy (DISE) classification system should cover all the upper airways, be simple and practical, and quantify the severity of any obstruction. Excellent validity and reliability are essential. We explored the inter-rater reliability of Koo's DISE classification system in the hands of experienced and inexperienced otolaryngologists. Methods We retrospectively analyzed video images of 100 patients who underwent DISE examination in our hospital between 2015 and 2018. Three experienced and three inexperienced otolaryngologists reviewed and scored all images. We calculated the inter-rater reliabilities of the two groups of otolaryngologists. Results Independent of the extent of experience with DISE, detection of retropalatal obstructions (overall agreement: 0.87; kappa value: 0.60), and the degree of such obstructions (overall agreement: 0.67; kappa value: 0.52) were more consistent than were the detection of retrolingual obstructions (overall agreement: 0.61, kappa value: 0.37) and the degree of retrolingual obstructions (overall agreement: 0.20, kappa value: 0.35). Inexperienced observers were in good agreement for palatal obstructions and experienced observers were in good agreement for tongue-base obstructions. All of the otolaryngologists found it difficult to detect a lateral pharyngeal wall obstruction at the retrolingual level. Conclusion Koo's DISE classification system focuses on surgical treatment, especially by otolaryngologists, and the degree of agreement between the experienced and inexperienced observers was relatively high. The participants' level of experience had a strong impact on scoring. The less-experienced otolaryngologists tended to overlook tongue-base obstructions, focusing instead on relatively simple retropalatal obstructions. In the future, development of a DISE classification system that can be accepted globally will be necessary.