Methods used to evaluate usability of mobile clinical decision support systems for healthcare emergencies: a systematic review and qualitative synthesis

Lay Summary Healthcare professionals must make safe, accurate decisions, especially during medical emergencies. Researchers design and develop tools that can help medical experts make these decisions. These tools are called Clinical Decision Support Systems (CDSSs). CDSSs obtain and process information about a patient, and display information to the healthcare professional (user) to aid decision-making. Whether the user finds the system easy to use or useful is referred to the system's usability. Usability affects how likely the CDSS is to be adopted and implemented into practice. We carefully searched the published literature and found 23 papers which measured the usability of CDSSs designed for medical emergencies. We found that CDSSs' efficiency and usefulness were measured the most, and effectiveness and memorability the least. More studies used questionnaires and user testing than interviews or specific "heuristic" evaluations. However, we found that interviews and heuristic evaluations identified more usability issues than did the questionnaires and user tests. Studies which tested the usability of CDSS by using both numerical methods (quantitative) and narrative methods (qualitative) were better at identifying the most issues. We advised both numerical and narrative methods to test the usability of CDSS, because it will be most comprehensive. Objective The aim of this study was to determine the methods and metrics used to evaluate the usability of mobile application Clinical Decision Support Systems (CDSSs) used in healthcare emergencies. Secondary aims were to describe the characteristics and usability of evaluated CDSSs. Materials and Methods A systematic literature review was conducted using Pubmed/Medline, Embase, Scopus, and IEEE Xplore databases. Quantitative data were descriptively analyzed, and qualitative data were described and synthesized using inductive thematic analysis. Results Twenty-three studies were included in the analysis. The usability metrics most frequently evaluated were efficiency and usefulness, followed by user errors, satisfaction, learnability, effectiveness, and memorability. Methods used to assess usability included questionnaires in 20 (87%) studies, user trials in 17 (74%), interviews in 6 (26%), and heuristic evaluations in 3 (13%). Most CDSS inputs consisted of manual input (18, 78%) rather than automatic input (2, 9%). Most CDSS outputs comprised a recommendation (18, 78%), with a minority advising a specific treatment (6, 26%), or a score, risk level or likelihood of diagnosis (6, 26%). Interviews and heuristic evaluations identified more usability-related barriers and facilitators to adoption than did questionnaires and user testing studies. Discussion A wide range of metrics and methods are used to evaluate the usability of mobile CDSS in medical emergencies. Input of information into CDSS was predominantly manual, impeding usability. Studies employing both qualitative and quantitative methods to evaluate usability yielded more thorough results. Conclusion When planning CDSS projects, developers should consider multiple methods to comprehensively evaluate usability.