Optimising the reproducibility of a performance-based assessment test in midwifery education

Despite problems concerning generalisability of test results - largely due to limited sampling of the task domain - and questions about test efficiency, the popularity of OSCEs in medical education has motivated the use of similar assessment methods in other health care domains. Purpose of this study was to investigate reproducibility of scores on an OSCE-based test in the relatively narrow domain of midwifery, as compared to the broad medical domain. The influence of global rating scales and adoption of a mastery-oriented test perspective on reproducibility of test scores was investigated in order to explore possibilities to increase test efficiency. A 3-hr, 6-station performance based test was administered to third and fourth year students at Kerkrade Midwifery School (the Netherlands). Students' performance was recorded using a station specific checklist and a global rating scale. For the 3-hr OSCE the generalisability coefficient based on checklist scores is 0.48. Based on global ratings alone, the generalisability coefficient is 0.61. Adjusted dependability indices are 0.85 based on checklist scores and 0.95 for global ratings respectively. Results suggest that even for small domains the problem of case-specificity remains a major impediment to high stakes performance testing. Findings furthermore suggest that use of global rating scales and professional expert judgements in professional competence evaluation is to be preferred to task specific checklists for reasons of reproducibility and efficiency in test development and administration. Adoption of a mastery oriented test perspective may reduce testing time requirements even further.