Validity of scores generated by a web-based multimedia simulated patient case software: A pilot study

Purpose. The value of multimedia simulated patient cases (MSPCs) in medical education remains unclear. The authors conducted a pilot study to assess the validity of automated scores of diagnostic reasoning ability provided by DxR Clinician(TM), a widely available Web-based MSPC software. Method. In 2002-03, all 89 students enrolled in a required third-year primary care clerkship at the University of California, Davis, School of Medicine were assigned to complete four MSPCs. The authors determined the degree of correlation between the Clinical Reasoning Score (CRS) and Level of Diagnostic Performance (LDP) generated by the MSPC software and subscale scores from a validated measure of diagnostic reasoning sophistication, the Diagnostic Thinking Inventory (DTI). Results. Of 356 completed case events, instructor override of automated scoring was required in 206 (58%) to obtain an accurate LDP and CRS. Mean DTI subscale scores improved significantly from the beginning to the end of the year (p < .0001, Wilcoxon signed rank test). However, there were no significant correlations between CRS or LDP scores on any of the four cases and either of the two DTI subscale scores. Conclusion. Automated diagnostic reasoning scores generated by one widely available MSPC software appear to lack criterion validity. The validity of automated diagnostic reasoning scores generated by MSPCs should be established before such cases can be confidently employed as educational tools.