The Medical Council of Canada (MCC) administers a qualifying examination for the issuance of a license to practice medicine. To date, this examination does not test the clinical skills of history taking, physical examination, and communication. The MCC is implementing an objective structured clinical examination (OSCE) to test these skills in October 1992. A pilot examination was developed to test the feasibility, reliability, and validity of running a multisite, two-form, four-hour, 20-station OSCE for national licensure. In February 1991, 240 volunteer first- and second-year residents were tested at four sites. The candidates were randomly assigned to one of two forms of the test and one of two sites for two of the four sites. Generalizability analysis revealed that the variance due to form was 0.0 and that due to site was .16 compared with a total variance of 280.86. The reliabilities (inter-station) were .56 and .60 for the two forms. Station total-test score correlations, used to measure station validity, were significant for 38 of the 40 stations used (range .14-.60). The results of the OSCE correlated moderately with the MCC qualifying examination; these correlations were .32 and .35 for the two test forms. Content validity was assessed by postexamination questionnaires given to the physician examiners using a scale of 0 (low) to 10 (high). The physicians' mean ratings were: importance of the stations, 8.1 (SD, 1.8); success of the examination in testing core skills, 8.1 (SD, 1.6); and degree of challenge, 7.8 (SD, 2.1). The results indicate that a full-scale national administration of an OSCE for licensure is feasible using the model developed. Aspects of validity have been established and strategies to augment reliability have been developed.
