Methodology to Quantify Accuracy for Procedure Execution Analysis

Several industries have sought to identify, mitigate, and predict human errors, as human errors contribute to accidents. We present a new methodology, Procedure Deviation Analysis (PDA), which establishes a human-error taxonomy for repair tasks. Unlike other approaches, PDA defines errors as deviations from the task procedure and considers all operator actions that differ from the procedure to classify and quantify procedure execution accuracy. PDA provides a quantitative measure of accuracy that yields insights for training efficacy and procedure design due to its broader consideration of error. PDA was developed to assess accuracy during repair tasks and to provide comparisons between training techniques. Two main findings of the PDA methodology were the establishment of six deviation modes and the associated application rules. To establish the PDA methodology, eighteen subjects participated in a two-session study and received one of two refresher trainings during the second session. The results indicated that neither refresher training prevented significant performance degradation. The total deviation mode occurrence significantly increased from the first to second session (p = 0.001). The average number of steps also increased (p = 0.036), due to subjects repeating steps. The percent accuracy significantly decreased from 87.07% to 61.07% (p < 0.0001), indicating that knowledge loss from the first to second session that was not adequately addressed in the refresher trainings. Furthermore, PDA highlighted potential areas for improvement in the procedures. PDA provided insights into both training efficacy and procedure quality, two of the key tools used to prevent negative outcomes in high-risk environments.