Validation of a Machine Learning Model for Early Shock Detection

Objectives: The objectives of this study were to test in real time a Trauma Triage, Treatment, and Training Decision Support (4TDS) machine learning (ML) model of shock detection in a prospective silent trial, and to evaluate specificity, sensitivity, and other estimates of diagnostic performance compared to the gold standard of electronic medical records (EMRs) review. Design: We performed a single-center diagnostic performance study. Patients and setting: A prospective cohort consisted of consecutive patients aged 18 years and older who were admitted from May 1 through September 30, 2020 to six Mayo Clinic intensive care units (ICUs) and five progressive care units. Measurements and main results: During the study time, 5,384 out of 6,630 hospital admissions were eligible. During the same period, the 4TDS shock model sent 825 alerts and 632 were eligible. Among 632 hospital admissions with alerts, 287 were screened positive and 345 were negative. Among 4,752 hospital admissions without alerts, 78 were screened positive and 4,674 were negative. The area under the receiver operating characteristics curve for the 4TDS shock model was 0.86 (95% CI 0.85-0.87%). The 4TDS shock model demonstrated a sensitivity of 78.6% (95% CI 74.1-82.7%) and a specificity of 93.1% (95% CI 92.4-93.8%). The model showed a positive predictive value of 45.4% (95% CI 42.6-48.3%) and a negative predictive value of 98.4% (95% CI 98-98.6%). Conclusions: We successfully validated an ML model to detect circulatory shock in a prospective observational study. The model used only vital signs and showed moderate performance compared to the gold standard of clinician EMR review when applied to an ICU patient cohort.