Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages

被引：81

作者：

Garcia, Patricia ^{[1
,3
]}

Ma, Stephen P. ^{[2
,3
]}

Shah, Shreya ^{[3
,4
]}

Smith, Margaret ^{[4
]}

Jeong, Yejin ^{[4
]}

Devon-Sand, Anna ^{[4
]}

Tai-Seale, Ming ^{[5
]}

Takazawa, Kevin ^{[6
]}

Clutter, Danyelle ^{[6
]}

Vogt, Kyle ^{[6
]}

Lugtu, Carlene ^{[7
]}

Rojo, Matthew ^{[6
]}

Lin, Steven ^{[3
,4
]}

Shanafelt, Tait ^{[3
,8
]}

Pfeffer, Michael A. ^{[3
,6
]}

Sharp, Christopher ^{[3
]}

机构：

[1] Stanford Univ, Dept Med, Sch Med, 430 Broadway St,3rd Floor, Redwood City, CA 94063 USA

[2] Stanford Univ, Dept Med, Sch Med, 453 Quarry Rd, Palo Alto, CA 94304 USA

[3] Stanford Univ, Sch Med, Dept Med, Stanford, CA USA

[4] Stanford Univ, Sch Med, Stanford Healthcare AI Appl Res Team, Div Primary Care & Populat Hlth, Stanford, CA USA

[5] Univ Calif San Diego, Sch Med, Dept Family Med, La Jolla, CA USA

[6] Stanford Med, Technol & Digital Solut, Stanford, CA USA

[7] Stanford Healthcare, Nursing Informat & Innovat, Stanford, CA USA

[8] Stanford Univ, WellMD Ctr, Sch Med, Stanford, CA USA

来源：

JAMA NETWORK OPEN | 2024年 / 7卷 / 03期

关键词：

PHYSICIANS; IMPACT;

D O I：

10.1001/jamanetworkopen.2024.3201

中图分类号：

R5 [内科学];

学科分类号：

1002 ; 100201 ;

摘要：

Importance The emergence and promise of generative artificial intelligence (AI) represent a turning point for health care. Rigorous evaluation of generative AI deployment in clinical practice is needed to inform strategic decision-making. Objective To evaluate the implementation of a large language model used to draft responses to patient messages in the electronic inbox. Design, Setting, and ParticipantsA 5-week, prospective, single-group quality improvement study was conducted from July 10 through August 13, 2023, at a single academic medical center (Stanford Health Care). All attending physicians, advanced practice practitioners, clinic nurses, and clinical pharmacists from the Divisions of Primary Care and Gastroenterology and Hepatology were enrolled in the pilot. InterventionDraft replies to patient portal messages generated by a Health Insurance Portability and Accountability Act-compliant electronic health record-integrated large language model. Main Outcomes and MeasuresThe primary outcome was AI-generated draft reply utilization as a percentage of total patient message replies. Secondary outcomes included changes in time measures and clinician experience as assessed by survey. ResultsA total of 197 clinicians were enrolled in the pilot; 35 clinicians who were prepilot beta users, out of office, or not tied to a specific ambulatory clinic were excluded, leaving 162 clinicians included in the analysis. The survey analysis cohort consisted of 73 participants (45.1%) who completed both the presurvey and postsurvey. In gastroenterology and hepatology, there were 58 physicians and APPs and 10 nurses. In primary care, there were 83 physicians and APPs, 4 nurses, and 8 clinical pharmacists. The mean AI-generated draft response utilization rate across clinicians was 20%. There was no change in reply action time, write time, or read time between the prepilot and pilot periods. There were statistically significant reductions in the 4-item physician task load score derivative (mean [SD], 61.31 [17.23] presurvey vs 47.26 [17.11] postsurvey; paired difference, -13.87; 95% CI, -17.38 to -9.50; P < .001) and work exhaustion scores (mean [SD], 1.95 [0.79] presurvey vs 1.62 [0.68] postsurvey; paired difference, -0.33; 95% CI, -0.50 to -0.17; P < .001). Conclusions and RelevanceIn this quality improvement study of an early implementation of generative AI, there was notable adoption, usability, and improvement in assessments of burden and burnout. There was no improvement in time. Further code-to-bedside testing is needed to guide future development and organizational strategy.

引用

页数：13

共 26 条

[11] Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models [J].