An assessment tool for brachial plexus regional anesthesia performance: Establishing construct validity and reliability

Background and Objectives: Technical proficiency in regional anesthesia is often determined subjectively through in-training evaluations. Objective assessment tools improve these evaluations by providing criteria for measurement. However, any evaluation instrument needs to be valid and reliable before it is adopted into a curriculum. The purpose of this study is to determine the validity and reliability of a devised assessment of residents performing, an interscalene brachial plexus block (ISB). Methods: In this prospective study, 10 junior trainees and 10 senior trainees were videotaped performing ail ISB. Junior trainees were defined as in their first year of anesthetic training and had performed less than 10 ISBs independently. Senior trainees had completed at least 1 year of anesthesia training and had performed greater than 10 ISBs independently. Two blinded expert raters independently evaluated the performance of the ISB using a checklist and global rating scale. Construct validity was established it the assessments were able to reliably discriminate between different levels of training. Results: Senior trainees performed an ISB significantly better than junior trainees when assessed using the global rating scale (P < .05) and checklist (P < .001). The overall interrater reliability for the global rating scores was excellent (r = 0.85, P < .05) and was good for the checklist scores (r = 0.74, P < .05). Conclusions: Both assessment modalities were valid, in that they reliably discriminated between different levels of training. Objective measures of technical skills are feasible, timely, and improve the validity and reliability of competency assessments.