Using Evidence to Improve Evaluation: A Comprehensive Psychometric Assessment of a SP-Based OSCE Licensing Examination

Background: Bridging the theory of psychometric assessment to an evaluation instrument is a challenging and continuous task. Such a task was completed for the SP-based OSCE licensing examination used in Quebec since 1990.Purpose: To review the comprehensive series of psychometric studies undertaken on the examination based on the theory of psychometric assessment. Methods: The examination is a 5.5 hour-long OSCE composed of 26 clinical cases of7, 10 or 15 minutes' duration. Candidates' total scores and case scores are used for licensing decisions. Reliability coefficients were obtained as well as generalizability studies to assess the impact of examination sites (4), tracks (14) and languages(French and English) on candidates' results. Its content validity was assessed by experts as well as by practicing family physicians, examiners and exam coordinators. Concurrent validity was assessed. Through differential item functioning (DIF)statistical approach, possible gender biases were analyzed. The predictive validity of the examination scores on selected aspects of practice was analyzed as well as the impact of the examination on the teaching and learning process.Results: These studies confirmed the validity, reliability and generalizability of the examination scores. No biases were demonstrated. Examination scores predicted elements of practice. Educational impact was present.Conclusion: This study confirmed the quality of the examination, ensured its continuous improvement and supported the inferences made from the examination results.