Automated Metrics in a Virtual-Reality Myringotomy Simulator: Development and Construct Validity

Objectives:The objectives of this study were: 1) to develop and implement a set of automated performance metrics into the Western myringotomy simulator, and 2) to establish construct validity.Study Design:Prospective simulator-based assessment study.Setting:The Auditory Biophysics Laboratory at Western University, London, Ontario, Canada.Participants:Eleven participants were recruited from the Department of Otolaryngology-Head & Neck Surgery at Western University: four senior otolaryngology consultants and seven junior otolaryngology residents.Interventions:Educational simulation.Main Outcome Measure:Discrimination between expert and novice participants on five primary automated performance metrics: 1) time to completion, 2) surgical errors, 3) incision angle, 4) incision length, and 5) the magnification of the microscope.Methods:Automated performance metrics were developed, programmed, and implemented into the simulator. Participants were given a standardized simulator orientation and instructions on myringotomy and tube placement. Each participant then performed 10 procedures and automated metrics were collected. The metrics were analyzed using the Mann-Whitney U test with Bonferroni correction.Results:All metrics discriminated senior otolaryngologists from junior residents with a significance of p<0.002. Junior residents had 2.8 times more errors compared with the senior otolaryngologists. Senior otolaryngologists took significantly less time to completion compared with junior residents. The senior group also had significantly longer incision lengths, more accurate incision angles, and lower magnification keeping both the umbo and annulus in view.Conclusions:Automated quantitative performance metrics were successfully developed and implemented, and construct validity was established by discriminating between expert and novice participants.