Does direct observation of procedural skills reflect trainee's progress in otolaryngology?

Background UK surgical trainees are required to undertake work-based assessments each year in order to progress in their training. Direct Observation of Procedural Skills (DOPS) is one of these assessments. We aim to investigate the validity of DOPS in assessing otolaryngology trainees at all levels. Methods A retrospective search of the portfolios of all otolaryngology trainees in North Thames was carried out to identify otolaryngology-specific DOPS. A score (Cs) was calculated for each DOPS based on the percentage of satisfactorily-rated items. The overall performance rating (Ps) was analysed as a separate variable and compared with Cs. The Ps and Cs results were then compared across trainee grades and levels within each grade: Core trainees (CT1-CT2) and specialty trainees (ST3-ST8). Results Seven hundred and sixty-seven otolaryngology DOPS were completed between August 2008 and September 2013. The tool was found to be reliable and internally consistent. Trainees in ST grade had higher Cs and Ps scores than CT grade (P<0.001). Pairwise comparison showed that both Cs and Ps increased from CT1 to ST3 (P=0.005) but not from ST4 onwards (P=0.198). Conclusions Otolaryngology DOPS is a useful tool in assessing otolaryngology trainees especially from CT1-ST3 level. DOPS can also differentiate between junior and senior trainees. However, it was not able to demonstrate progress at levels above ST3, most likely due to the simplicity of the procedures which trainees tend to master in the first few years of training.