The effect of incorporating normative data into a criterion-referenced standard setting in medical education

Purpose. Determining standards for assessing clinical performance is a controversial issue. Purely item-based methods such as the Angoff method often produce unrealistic judgments, even when used by experienced judges. The rather unstudied compromise methods combine absolute and relative judgments and thereby incorporate normative data into criterion-based standard-setting judgments. The purpose of this study was to compare the quality and implications of standards set by three methods used for the Objective Structured Clinical Examination (OSCE). Method. Ninety-six judges set standards for 36 surgical year-4 undergraduate OSCE stations. All judges had normative student performance data when judgments were made with the Angoff, Ebel, or Hofstee methods. Results. The Hofstee, method gave more realistic cutoff scores and standard errors and better Meskauskas and Jaeger indices than the Angoff and Ebel methods. Conclusions. Medical educators setting standards for an OSCE should consider adopting the Hofstee method.