Reliability of a pediatric sleep endoscopy scoring system

Introduction: The surgical management of Obstructive Sleep Apnea (OSA) may be guided by Drug Induced Sleep Endoscopy (DISE), but there is no universally accepted scoring system for DISE in children. The purpose of this study is to compare a novel system to the more commonly used VOTE (Velum, Oropharynx, Tongue base, Epiglottis) system. Methods: A total of thirty pediatric DISE videos were reviewed and scored twice by 5 raters with a range of experience levels. The videos were scored using both the VOTE and the new scoring systems. The raters were also asked to recommend surgical intervention based on the DISE exam alone. Intra-rater test-retest analysis of the responses was conducted using weighted kappa (WK) statistic and percentage agreement (PA), respectively. Inter-rater reliability analysis of responses was evaluated using Krippendorff's alpha reliability coefficient (KA). Using a proportional odds model, a comparison of the weighted kappa statistic for the VOTE and the novel scoring systems was conducted. Results: For the novel scoring system, the intra-rater test-retest WK coefficient was 0.62 and 0.87 and the PA was 64% and 82% at the 25th and 75th percentiles, respectively. For the VOTE system, the intra-rater test-retest WK coefficient was 0.50 and 1.00 and PA was 75% and 100% at the 25th and 75th percentiles, respectively. KA was 0.36 and 0.77 at the 25th and 75th percentiles. In addition, responses to the surgical intervention survey analysis yielded a test-retest WK coefficient of 0.40 and 1.00 and a PA 71%-100% at the 25th and 75th percentiles, respectively. KA was 0.40 and 0.61 at the 25th and 75th percentiles, respectively. There was no significant difference between the 2 scoring systems on the WK statistic (p = .15), but significant difference on PA (p = .01). Conclusion: Our novel scoring system demonstrated intra-rater test-retest and inter-rater reliability similar to the VOTE system, includes more sites of potential airway obstruction, and has potential for widespread use.