Development, deployment, and continuous monitoring of a machine learning model to predict respiratory failure in critically ill patients

Objectives This study describes the development and deployment of a machine learning (ML) model called Vent.io to predict mechanical ventilation (MV).Materials and Methods We trained Vent.io using electronic health record data of adult patients admitted to the intensive care units (ICUs) of the University of California San Diego (UCSD) Health System. We prospectively deployed Vent.io using a real-time platform at UCSD and evaluated the performance of Vent.io for a 1-month period in silent mode and on the MIMIC-IV dataset. As part of deployment, we included a Predetermined Changed Control Plan (PCCP) for continuous model monitoring that triggers model fine-tuning if performance drops below a specified area under the receiver operating curve (AUC) threshold of 0.85.Results The Vent.io model had a median AUC of 0.897 (IQR: 0.892-0.904) with specificity of 0.81 (IQR: 0.812-0.841) and positive predictive value (PPV) of 0.174 (IQR: 0.148-0.176) at a fixed sensitivity of 0.6 during 10-fold cross validation and an AUC of 0.908, sensitivity of 0.632, specificity of 0.849, and PPV of 0.235 during prospective deployment. Vent.io had an AUC of 0.73 on the MIMIC-IV dataset, triggering model fine-tuning per the PCCP as the AUC was below the minimum of 0.85. The fine-tuned Vent.io model achieved an AUC of 0.873.Discussion Deterioration of model performance is a significant challenge when deploying ML models prospectively or at different sites. Implementation of a PCCP can help models adapt to new patterns in data and maintain generalizability.Conclusion Vent.io is a generalizable ML model that has the potential to improve patient care and resource allocation for ICU patients with need for MV. Earlier identification of patients at the highest risk of requiring mechanical ventilation (MV) offers an opportunity for timely medical interventions and efficient resource allocation. In this study, we developed a machine learning (ML) model called Vent.io for the prediction of MV up to 24 hours in advance using a combination of vital signs, laboratory measurements, comorbidities, medications, and demographic features. We trained Vent.io using intensive care unit (ICU) data from the University of California San Diego (UCSD) Health System and deployed it in our real-time predictive analytics platform with a Predetermined Changed Control Plan (PCCP) for continuous model monitoring that triggers model fine-tuning if performance drops below a specified area under the receiver operating curve (AUC) threshold of 0.85. Vent.io was prospectively validated after 1 month of deployment at UCSD and achieved an AUC of 0.908, meaning no model fine-tuning was required. We then simulated local deployment of Vent.io at an external site by evaluating Vent.io on the MIMIC-IV dataset. The resulting AUC of 0.73 was below the PCCP threshold, so Vent.io was fine-tuned using MIMIC-IV data, resulting in an AUC of 0.873. These results show Vent.io is generalizable and can aid clinicians in identifying high-risk patients for MV.