Clinical skills training: developing objective assessment instruments

The purpose of this study was to develop objective assessment instruments for use in psychomotor skill training and to test the instruments for interobserver reliability. Two checklist style instruments, one for suturing and one for endotracheal intubation, were developed through a process of review of standard texts, consultation with local experts and field testing. Following development they were used by paired examiners in an Objective Structured Clinical Examination (OSCE) setting to test the instruments for interobserver reliability. A total of 88 final year medical students were recruited from the five Ontario medical schools to participate as examinees. The checklists worked well within the practical constraints of a 10 minute OSCE station and demonstrated a high level of interobserver reliability with Kappa scores of 0.65 for the suturing checklist and 0.71 for the intubation checklist. Furthermore, the Kappa scores for individual checklist items served to identify items which demonstrated poor interobserver reliability and thus highlighted them for review.