Multimodal deep learning for predicting in-hospital mortality in heart failure patients using longitudinal chest X-rays and electronic health records

Amid an aging global population, heart failure has become a leading cause of hospitalization among older people. Its high prevalence and mortality rates underscore the importance of accurate mortality prediction for swift disease progression assessment and better patient outcomes. The evolution of artificial intelligence (AI) presents new avenues for predicting heart failure mortality. Yet current research has predominantly leveraged structured data and unstructured clinical notes from electronic health records (EHR), underutilizing the prognostic value of chest X-rays (CXRs). This study aims to harness deep learning methodologies to explore the feasibility of enhancing the precision of predicting in-hospital all-cause mortality in heart failure patients using CXRs data. We propose a novel multimodal deep learning network based on the spatially and temporally decoupled Transformer (MN-STDT) for in-hospital mortality prediction in heart failure by integrating longitudinal CXRs and structured EHR data. The MN-STDT captures spatial and temporal information from CXRs through a Hybrid Spatial Encoder and a Distance-Aware Temporal Encoder, ultimately fusing features from both modalities for predictive modeling. Initial pre-training of the spatial encoder was conducted on CheXpert, followed by full model training and evaluation on the MIMIC-IV and MIMIC-CXR datasets for mortality prediction tasks. In a comprehensive view, the MN-STDT demonstrated the best performance, with an AUC-ROC of 0.8620, surpassing all baseline models. Comparative analysis revealed that the AUC-ROC of the multimodal model (0.8620) was significantly higher than that of models using only structured data (0.8166) or chest X-ray data alone (0.7479). This study demonstrates the value of CXRs in the prognosis of heart failure, showing that the combination of longitudinal CXRs with structured EHR data can significantly improve the accuracy of mortality prediction in heart failure. Feature importance analysis based on SHAP provides interpretable decision support, paving the way for potential clinical applications.