Measuring outcomes of undergraduate medical education: Residency directors' ratings of first-year residents

Purpose. There is a growing recognition of the need to show the relationship between undergraduate medical education (UME) and achievements during residency. This study provides reliability and validity evidence for a residency rating scale as well as a method for gathering comparison information about first-year residents. Method. A 25-item rating scale measuring important areas of physician functioning was mailed to residency directors of 485 graduates of the 1998-2000 classes of the University of Kansas School of Medicine. The same rating scale was sent to residency directors for a comparison sample of 251 graduates of other U.S. medical schools who were residents at the University of Kansas Medical Center. Each item, on the rating scale was rated on a five-point Likert scale. Principal-components analysis, correlational analyses, internal consistency reliability analysis, and mean comparisons were used to provide evidence of reliability and validity. Results. A total of 382 (82%) usable rating scales were returned. A principal-components analysis extracted five factors that accounted for 86% of the variance. The final factors were (1) interpersonal communication, (2) clinical skills, (3) population-based health care, (4) record-keeping skills, and (5) critical appraisal skills. The internal consistency of the entire scale was .98, with coefficients for the five factors ranging from .92 to .97. The correlations between the five factors and measures of undergraduate performance ranged from .21 to .49. Group analyses revealed that residents with high GPAs and USMLE Step I and Step 2 scores tended to be rated higher than those with lower scores. Conclusions. The rating scale demonstrated adequate reliability and validity and showed that residency directors' ratings are a useful outcome measure for UME performance.