Development and external validation of multimodal postoperative acute kidney injury risk machine learning models

Objectives To develop and externally validate machine learning models using structured and unstructured electronic health record data to predict postoperative acute kidney injury (AKI) across inpatient settings.Materials and Methods Data for adult postoperative admissions to the Loyola University Medical Center (2009-2017) were used for model development and admissions to the University of Wisconsin-Madison (2009-2020) were used for validation. Structured features included demographics, vital signs, laboratory results, and nurse-documented scores. Unstructured text from clinical notes were converted into concept unique identifiers (CUIs) using the clinical Text Analysis and Knowledge Extraction System. The primary outcome was the development of Kidney Disease Improvement Global Outcomes stage 2 AKI within 7 days after leaving the operating room. We derived unimodal extreme gradient boosting machines (XGBoost) and elastic net logistic regression (GLMNET) models using structured-only data and multimodal models combining structured data with CUI features. Model comparison was performed using the receiver operating characteristic curve (AUROC), with Delong's test for statistical differences.Results The study cohort included 138 389 adult patient admissions (mean [SD] age 58 [16] years; 11 506 [8%] African-American; and 70 826 [51%] female) across the 2 sites. Of those, 2959 (2.1%) developed stage 2 AKI or higher. Across all data types, XGBoost outperformed GLMNET (mean AUROC 0.81 [95% confidence interval (CI), 0.80-0.82] vs 0.78 [95% CI, 0.77-0.79]). The multimodal XGBoost model incorporating CUIs parameterized as term frequency-inverse document frequency (TF-IDF) showed the highest discrimination performance (AUROC 0.82 [95% CI, 0.81-0.83]) over unimodal models (AUROC 0.79 [95% CI, 0.78-0.80]).Discussion A multimodality approach with structured data and TF-IDF weighting of CUIs increased model performance over structured data-only models.Conclusion These findings highlight the predictive power of CUIs when merged with structured data for clinical prediction models, which may improve the detection of postoperative AKI. Acute kidney injury (AKI) after an operation, called postoperative AKI, is common in hospitalized patients and associated with increased morbidity and mortality. Early detection of high-risk patients could facilitate timely treatment and improve outcomes. Although a few studies have developed machine learning (ML) models to identify patients with postoperative AKI, these are primarily limited to structured data (eg, laboratory values) and ignore predictors from clinical notes. Further, models built from clinical notes are often not externally validated because doing so risks leaking protected health information.Given these limitations in the field, we developed and externally validated ML models to predict postoperative AKI using structured data and information from clinical notes. To preserve patient privacy, we used concept unique identifiers (CUIs), which are de-identified medical terms from clinical notes. We compared unimodal models with structured data to multimodal models with CUIs plus structured data, as well as different approaches to modeling the CUI data. We found that multimodal models significantly improved model performance compared to unimodal models. We also found that normalizing CUI data based on term frequency had the highest performance. In conclusion, using CUIs to account for information in clinical notes adds significant value for predicting postoperative AKI.