Practice makes better? Testing a model for training program evaluators in situation awareness

Evaluation approaches should be appropriate for their contexts. Scholars and practitioners alike have widely acknowledged this view for at least four decades, and the ability to respond to context is clearly established as a core evaluator competency. Outside of evaluation, this knowledge of context, known as situation awareness, is seen as a critical feature of good decision making and as a factor that distinguishes experts from novices across a wide range of domains. Yet there are few opportunities for evaluators to explicitly build their skills in situation awareness. This study addressed that gap by examining the potential for one online training program, informed by research on deliberate practice, to accelerate evaluators' progress towards expertise in situation awareness. Built to align with evidence on developing situation awareness skills, the EvalPractice portal combined (1) an extensive 'case bank' of real-world evaluation scenarios, (2) repetitive practice that allowed novice evaluators to practice interpreting these evaluation scenarios, and (3) immediate feedback on the accuracy of these efforts based on events from the real-life version of the scenario. Findings from a small EvalPractice pilot suggest that it may be possible to improve foundational situation awareness skills using deliberate practice, but that further research is required to understand strategies for building higher-level situation awareness skills in evaluation.