High-fidelity patient simulation: validation of performance checklists

Background. Standardized scenarios can be used for performance assessments geared to the level of the learner. The purpose of this study was to validate checklists used for the assessments of medical students' performance using high-fidelity patient simulation. Methods. Our undergraduate committee designed 10 scenarios based on curriculum objectives. Fifteen faculty members with undergraduate educational experience identified items considered appropriate for medical students' performance level and identified items that, if omitted, would negatively affect grades. Items endorsed by less than 20% of faculty were omitted. For remaining items, weighting was calculated according to faculty responses. Students managed at least one scenario during which their performance was videotaped. Two raters independently completed the checklists for three consecutive sessions to determine inter-rater reliability. Validity was determined using Cronbach's alpha with an alphagreater than or equal to0.6 and less than or equal to0.9 considered acceptable internal consistency. Item analysis was performed by recalculating Cronbach's alpha with each item deleted to determine if that item contributed to a low internal consistency. Results. 135 students participated in the study. Inter-rater reliability of the two raters determined on the third session was 0.97 and therefore one rater completed the remaining performance assessments. Cronbach's alpha for the 10 scenarios ranged from 0.16 to 0.93 with two scenarios demonstrating acceptable internal consistency with all items. Three scenarios demonstrated acceptable internal consistency with one item deleted. Conclusions. Five scenarios developed for this study were shown to be valid when using the faculty criteria for expected performance level.