Design, Validity, and Reliability of a Pediatric Resident JumpSTART Disaster Triage Scoring Instrument

OBJECTIVE: To design an instrument for scoring residents learning pediatric disaster triage (PDT), and to test the validity and reliability of the instrument. METHODS: We designed a checklist-based scoring instrument including PDT knowledge and skills and performance, as well as a global assessment. Learners' performance in a 10-patient school bus crash simulation was video recorded and scored with the instrument. Learners triaged the patients with a color-coded algorithm (JumpSTART, Simple Triage and Rapid Treatment). Three evaluators observed the recordings and scored triage performance for each learner. Internal and construct validity of the instrument were established via comparison of resident performance by postgraduate year (PGY) and correlating instrument items with overall score. Validity was assessed with analysis of variance and the D statistic. We calculated evaluators' intraclass correlation coefficient (ICC) for each patient, skill, triage decision, and global assessment. RESULTS: There were 37 learners and 111 observations. There was no difference in total scores by PGY (P=.77), establishing internal validity. Regarding construct validity, most instrument items had a D statistic of >0.5. The overall ICC among scores was 0.83 (95% confidence interval [CI] 0.74-0.89). Individual patient score reliability was high and was greatest among patients with head injury (ICC 0.86; 95% CI 0.79-0.91). Reliability was low for an ambulatory patient (ICC 0.29; 95% CI 0.07-0.48). Triage skills evaluation showed excellent reliability, including airway management (ICC 0.91; 95% CI 0.86-0.94) and triage speed (ICC 0.81; 95% CI 0.72-0.88). The global assessment had moderate reliability for skills (ICC 0.63; 95% CI 0.47-0.75) and knowledge (ICC 0.64; 95% CI 0.49-0.76). CONCLUSIONS: We report the validity and reliability testing of a PDT-scoring instrument. Validity was confirmed with no performance differential by PGY. Reliability of the scoring instrument for most patient-level triage, knowledge, and specific skills was high.