Comparison of Multimodal Deep Learning Approaches for Predicting Clinical Deterioration in Ward Patients: Observational Cohort Study

Background: Implementing machine learning models to identify clinical deterioration in the wards is associated with decreased morbidity and mortality. However, these models have high false positive rates and only use structured data. Objective: We aimed to compare models with and without information from clinical notes for predicting deterioration. Methods: Adults admitted to the wards at the University of Chicago (development cohort) and University of Wisconsin-Madison (external validation cohort) were included. Predictors consisted of structured and unstructured variables extracted from notes as concept unique identifiers (CUIs). We parameterized CUIs in 5 ways: standard tokenization (ST), International Classification of Diseases rollup using tokenization (ICDR-T), International Classification of Diseases rollup using binary variables (ICDR-BV), concept unique identifiers as SapBERT embedding (SE), and concept unique identifier clustering using SapBERT embeddings (CC). Each parameterization method combined with structured data and each structured data-only method were compared for predicting intensive care unit transfer or death in the next 24 hours using deep recurrent neural networks. Results: The development (University of Chicago) cohort included 284,302 patients, while the external validation (University of Wisconsin-Madison) cohort included 248,055 patients. In total, 4.9% (n=26,281) of patients experienced the outcome. The SE model achieved the highest area under the precision-recall curve (0.208), followed by CC (0.199) and the structured-only model (0.199), ICDR-BV (0.194), ICDR-T (0.166), and ST (0.158). The CC and structured-only models achieved the highest area under the receiver operating characteristic (0.870), followed by ICDR-T (0.867), ICDR-BV (0.866), ST (0.860), and SE (0.859). Regarding sensitivity and positive predictive value, the CC model achieved the greatest positive predictive value (12.53%) and sensitivity (52.15%) at the cutoff that flagged 5% of the observations in the test set. At the 15% cutoff, the ICDR-T, CC, and ICDR-BV models tied for the highest positive predictive value at 5.67%, while their sensitivities were 70.95%, 70.92%, and 70.86%, respectively. All models were well calibrated, achieving Brier scores in the range of 0.011-0.012. The modified integrated gradients method revealed that CUIs corresponding to terms such as "NPO-nothing by mouth," "chemotherapy," "transplanted tissue," and "dialysis procedure" were most predictive of deterioration. Conclusions: A multimodal model combining structured data with embeddings using SapBERT had the highest area under the precision-recall curve, but performance was similar between models with and without CUIs. Although the addition of CUIs from notes to structured data did not meaningfully improve model performance for predicting clinical deterioration, models using CUIs could provide clinicians with relevant information and additional clinical context for supporting decision-making.