The Effect of Evaluator Training on Inter- and Intrarater Reliability in High-Stakes Assessment in Simulation

AIM The aim of this study was to evaluate the effectiveness of a training intervention in achieving inter- A nd intrarater reliability among faculty raters conducting high-stakes assessment of clinical performance in simulation. BACKGROUND High-stakes assessment of simulation performance is being adopted in nursing education. However, limited research exists to guide best practices in training raters, which is essential to ensure fair and defensible assessment. METHOD A nationwide sample of 75 prelicensure RN program faculty participated in an experimental, randomized, controlled study. RESULTS Participants completing a training intervention achieved higher inter- A nd intrarater reliability than control group participants when using a checklist evaluation tool. Mixed results were achieved by participants when completing a global competency assessment. CONCLUSION The training intervention was effective in helping participants to achieve a shared mental model for use of a checklist, but more time may be necessary to achieve consistent global competence decisions. © 2020 Lippincott Williams and Wilkins.