Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study

Objective Many options are currently available for sepsis surveillance clinical decision support (CDS) from electronic medical record (EMR) vendors, third party, and homegrown models drawing on rule-based (RB) and machine learning (ML) algorithms. This study explores sepsis CDS implementation from the perspective of implementation leads by describing the motivations, tool choices, and implementation experiences of a diverse group of implementers. Materials and Methods Semi-structured interviews were conducted with and a questionnaire was administered to 21 hospital leaders overseeing CDS implementation at 15 US medical centers. Participants were recruited via convenience sampling. Responses were coded by 2 coders with consensus approach and inductively analyzed for themes. Results Use of sepsis CDS is motivated in part by quality metrics for sepsis patients. Choice of tool is driven by ease of integration, customization capability, and perceived predictive potential. Implementation processes for these CDS tools are complex, time-consuming, interdisciplinary undertakings resulting in heterogeneous choice of tools and workflow integration. To improve clinician acceptance, implementers addressed both optimization of the alerts as well as clinician understanding and buy in. More distrust and confusion was reported for ML models, as compared to RB models. Respondents described a variety of approaches to overcome implementation barriers; these approaches related to alert firing, content, integration, and buy-in. Discussion While there are shared socio-technical challenges of implementing CDS for both RB and ML models, attention to user education, support, expectation management, and dissemination of effective practices may improve feasibility and effectiveness of ML models in quality improvement efforts. Conclusion Further implementation science research is needed to determine real world efficacy of these tools. Clinician acceptance is a significant barrier to sepsis CDS implementation. Successful implementation of less clinically intuitive ML models may require additional attention to user confusion and distrust. Lay Summary Sepsis is a life-threatening illness. Improving sepsis care is a growing priority for many hospitals. Patients at risk of developing sepsis can be identified before they get very sick using tools that analyze data from computerized medical records systems. A variety of options are available from different sources. Some tools are programmed using established sepsis screening criteria used in clinical practice. Others rely on machine learning, where computer algorithms identify patterns in the available data without being pre-programmed by a human being. In this study, we interviewed 21 individuals at 15 US medical centers who oversaw hospital level implementations of these tools. Teams were motivated by wanting to improve quality of care for patients with sepsis. One major challenge was making the tools identify as many patients truly at risk for sepsis as possible while limiting false identification of patients not actually at risk. Many interviewees also described lack of trust in the tools from the nurses and doctors using the tools. There was more distrust and confusion reported by implementers of tools that relied on machine learning than tools that programmed human logic. Strategies emphasizing user education, user support, and expectation management were reported to be helpful.