Using AI-generated suggestions from ChatGPT to optimize clinical decision support

被引：200
作者：
Liu, Siru [1 ,8 ]
Wright, Aileen P. [1 ,2 ]
Patterson, Barron L. [3 ]
Wanderer, Jonathan P. [1 ,4 ]
Turer, Robert W. [5 ,6 ]
Nelson, Scott D. [1 ]
McCoy, Allison B. [1 ]
Sittig, Dean F. [7 ]
Wright, Adam [1 ]
机构：
[1] Vanderbilt Univ, Med Ctr, Dept Biomed Informat, Nashville, TN USA
[2] Vanderbilt Univ, Dept Med, Med Ctr, Nashville, TN USA
[3] Vanderbilt Univ, Dept Pediat, Med Ctr, Nashville, TN USA
[4] Vanderbilt Univ, Med Ctr, Dept Anesthesiol, Nashville, TN USA
[5] Univ Texas Southwestern Med Ctr, Dept Emergency Med, Dallas, TX USA
[6] Univ Texas Southwestern Med Ctr, Clin Informat Ctr, Dallas, TX USA
[7] Univ Texas Hlth Sci Ctr Houston, Sch Biomed Informat, Houston, TX USA
[8] Vanderbilt Univ, Med Ctr, Dept Biomed Informat, 2525 West End Ave 1475, Nashville, TN 37212 USA
来源：
JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION | 2023年 / 30卷 / 07期
关键词：
artificial intelligence; clinical decision support; large language model; ALERTS; DISPARITIES; SYSTEMS;
D O I：
10.1093/jamia/ocad072
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
Objective To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
引用
收藏
页码：1237 / 1245
页数：9
相关论文
共 50 条
[41]   Revolutionary Potential of ChatGPT in Constructing Intelligent Clinical Decision Support Systems [J].
Liao, Zhiqiang ;
Wang, Jian ;
Shi, Zhuozheng ;
Lu, Lintao ;
Tabata, Hitoshi .
ANNALS OF BIOMEDICAL ENGINEERING, 2024, 52 (02) :125-129
[42]   Explainability in medicine in an era of AI-based clinical decision support systems [J].
Pierce, Robin L. ;
Van Biesen, Wim ;
Van Cauwenberge, Daan ;
Decruyenaere, Johan ;
Sterckx, Sigrid .
FRONTIERS IN GENETICS, 2022, 13
[43]   Toward a responsible future: recommendations for AI-enabled clinical decision support [J].
Labkoff, Steven ;
Oladimeji, Bilikis ;
Kannry, Joseph ;
Solomonides, Anthony ;
Leftwich, Russell ;
Koski, Eileen ;
Joseph, Amanda L. ;
Lopez-Gonzalez, Monica ;
Fleisher, Lee A. ;
Nolen, Kimberly ;
Dutta, Sayon ;
Levy, Deborah R. ;
Price, Amy ;
Barr, Paul J. ;
Hron, Jonathan D. ;
Lin, Baihan ;
Srivastava, Gyana ;
Pastor, Nuria ;
Luque, Unai Sanchez ;
Bui, Tien Thi Thuy ;
Singh, Reva ;
Williams, Tayler ;
Weiner, Mark G. ;
Naumann, Tristan ;
Sittig, Dean F. ;
Jackson, Gretchen Purcell ;
Quintana, Yuri .
JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION, 2024, 31 (11) :2730-2739
[44]   From Algorithms to Academia: An Endeavor to Benchmark AI-Generated Scientific Papers against Human Standards [J].
Woodrow, Jackson ;
Nassour, Nour ;
Kwon, John Y. ;
Ashkani-Esfahani, Soheil ;
Harris, Mitchel .
ARCHIVES OF BONE AND JOINT SURGERY-ABJS, 2025, 13 (04) :212-222
[45]   Integration of AI in surgical decision support: improving clinical judgment [J].
Balch, Jeremy A. ;
Shickel, Benjamin ;
Bihorac, Azra ;
Upchurch Jr, Gilbert R. ;
Loftus, Tyler J. .
GLOBAL SURGICAL EDUCATION - JOURNAL OF THE ASSOCIATION FOR SURGICAL EDUCATION, 2024, 3 (01)
[46]   Using statistical anomaly detection models to find clinical decision support malfunctions [J].
Ray, Soumi ;
McEvoy, Dustin S. ;
Aaron, Skye ;
Hickman, Thu-Trang ;
Wright, Adam .
JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION, 2018, 25 (07) :862-871
[47]   Use of Simulation to Study Nurses' Acceptance and Nonacceptance of Clinical Decision Support Suggestions [J].
Sousa, Vanessa E. C. ;
Lopez, Karen Dunn ;
Febretti, Alessandro ;
Stifter, Janet ;
Yao, Yingwei ;
Johnson, Andrew ;
Wilkie, Diana J. ;
Keenan, Gail M. .
CIN-COMPUTERS INFORMATICS NURSING, 2015, 33 (10) :465-472
[48]   AI-enabled clinical decision support tools for mental healthcare: A product review [J].
Kleine, Anne-Kathrin ;
Kokje, Eesha ;
Hummelsberger, Pia ;
Lermer, Eva ;
Schaffernak, Insa ;
Gaube, Susanne .
ARTIFICIAL INTELLIGENCE IN MEDICINE, 2025, 160
[49]   APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support [J].
Kwong, Jethro C. C. ;
Khondker, Adree ;
Lajkosz, Katherine ;
Mcdermott, Matthew B. A. ;
Frigola, Xavier Borrat ;
Mccradden, Melissa D. ;
Mamdani, Muhammad ;
Kulkarni, Girish S. ;
Johnson, Alistair E. W. .
JAMA NETWORK OPEN, 2023, 6 (09)
[50]   Can You Spot the AI-Generated Images? Distinguishing Fake Images Using Signal Detection Theory [J].
Park, Hayun ;
Kim, Gayoung ;
Fee, Danbi ;
Kiw, Hyun K. .
CROSS-CULTURAL DESIGN, PT IV, CCD 2024, 2024, 14702 :299-313
12345