Anaesthesia Non-Technical Skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day?

Performance assessment is becoming increasingly necessary in the medical workplace. Hospitals and patients expect safety, and under-performance by a doctor can compromise standards. By describing and quantifying performance, positive behaviour can be encouraged and unsafe behaviour remedied. Anaesthesia Non-Technical Skills (ANTS) is a behavioural marker system that can be used to assess non-technical skills in the workplace. We determined whether specialist anaesthetists could be reliably trained to use ANTS at an assessor level in an 8 h programme. Unscripted videos of routine anaesthesia were produced for training and assessment purposes. Twenty-six participants attended rater training. Exercises in behaviour observation, rater error, frame of reference and performance dimension, and the use of ANTS were conducted throughout the day. Five videos were selected for formal assessment and data collected. Intra-class correlations (ICCs) were calculated for each element. The accepted value of r > 0.7 was not reached. ICC calculated for each element was 0.11-0.62. Comparison of participants scores with those of expert raters showed poor agreement. Anaesthetists could not be trained to reliably use ANTS as a summative assessment tool using our 1 day programme. There was an inadequate correlation of scores between participants and experts. Two major problems contributed to the lack of agreement. Observed behaviours were often misclassified into the incorrect element and safety beliefs varied among anaesthetists. Other reasons for the failure to achieve success and potential future direction are discussed.