Reliability of the welling scale (WS1) for rating temporal bone dissection performance

Objective: To determine the inter- and mitrarater reliability of using a new scale (Welling scale) for resident evaluation of temporal bone dissection performance. Study Design: Prospective, double-blinded, randomized trial. Methods: Twelve residents in otolaryngology (postgraduate year [PGY] 2-5) drilled 26 temporal I bones (21 cadaveric, 5 plastic) with the objective to perform a complete mastoidectomy with facial recess approach. These bones were then rated using the Welling scale by six independent raters on two separate occasions (4-6 wk apart). Raters were blinded to PGY year. The Kappa statistic was calculated for inter- and intrarater reliability. Results: Intrarater agreement was high for all raters, ranging from K = 0.65 to 0.72 (all P < .001), whereas the interrater agreement scores were more moderate (range, K = 0.49-0.64; all P < .01). Conclusion: The Welling scale can be used reliably to assess temporal bone dissection performance where performance is measured by assessment of end product (mastoidectomy with facial recess approach).