Low Correlation between Subjective and Objective Measures of Knowledge on Surgery Clerkships

BACKGROUND: Medical student knowledge is assessed during surgical clerkships subjectively and objectively. Subjective evaluation depends on faculty assessment during clinical and didactic interactions. Objective measurement derives from standardized tools, such as the National Board of Medical Examiners Surgery Subject test (shelf). Few efforts have been made to characterize the correlation between subjective and objective measures of medical knowledge. STUDY DESIGN: All 308 third-year medical students who completed the 8-week surgery clerkship at the University of North Carolina at Chapel Hill between July 2005 and June 2007 received subjective assessment of knowledge on 3 clinical rotations (one 4-week core and two 2-week elective rotations) and a longitudinal small-group tutorial. Faculty evaluators assigned percentile scores to rate students' knowledge base relative to their peers. In addition, students took the shelf test the last day of clerkship, and percentile scores were assigned based on National Board of Medical Examiners supplied normative data from first-time test-takers within the same academic quarter. Subjective versus objective knowledge scores were plotted overall, and Pearson product-moment correlation coefficients were generated for core, elective, and tutorial assessments. RESULTS: There were only weak linear relationships noted between subjective faculty-assigned knowledge scores and objective shelf scores. Pearson correlations were 0.24 for core rotations (4 weeks exposure), 0.14 for elective rotations (2 weeks exposure), and 0.22 for tutorials (1-hour exposure/week during 8 weeks), with p values <0.0001. CONCLUSIONS: Faculty assessment of knowledge is only weakly correlated with shelf performance. Faculty evaluations after 4-week rotations or longitudinal small-group interactions are better correlated with shelf scores than after 2-week electives. (J Am Coll Surg 2010;210:680-685. (C) 2010 by the American College of Surgeons)