Unveiling the risks of ChatGPT in diagnostic surgical pathologyChatGPT

被引:5
作者
Guastafierro, Vincenzo [1 ,2 ]
Corbitt, Devin N. [1 ]
Bressan, Alessandra [1 ,2 ]
Fernandes, Bethania [2 ]
Mintemur, Omer [2 ]
Magnoli, Francesca [3 ]
Ronchi, Susanna [3 ]
La Rosa, Stefano [3 ,4 ]
Uccella, Silvia [1 ,2 ]
Renne, Salvatore Lorenzo [1 ,2 ]
机构
[1] Humanitas Univ, Dept Biomed Sci, Via Rita Levi Montalcini 4, I-20072 Milan, Italy
[2] IRCCS Humanitas Res Hosp, Dept Pathol, Via Manzoni 56, I-20089 Milan, Italy
[3] ASST Sette Laghi, Dept Oncol, Unit Pathol, Varese, Italy
[4] Univ Insubria, Dept Med & Technol Innovat, Unit Pathol, Varese, Italy
关键词
ChatGPT; Surgical pathology; Large language model; Accuracy; Usefulness;
D O I
10.1007/s00428-024-03918-1
中图分类号
R36 [病理学];
学科分类号
100104 ;
摘要
ChatGPT, an AI capable of processing and generating human-like language, has been studied in medical education and care, yet its potential in histopathological diagnosis remains unexplored. This study evaluates ChatGPT's reliability in addressing pathology-related diagnostic questions across ten subspecialties and its ability to provide scientific references. We crafted five clinico-pathological scenarios per subspecialty, simulating a pathologist using ChatGPT to refine differential diagnoses. Each scenario, aligned with current diagnostic guidelines and validated by expert pathologists, was posed as open-ended or multiple-choice questions, either requesting scientific references or not. Outputs were assessed by six pathologists according to. (1) usefulness in supporting the diagnosis and (2) absolute number of errors. We used directed acyclic graphs and structural causal models to determine the effect of each scenario type, field, question modality, and pathologist evaluation. We yielded 894 evaluations. ChatGPT provided useful answers in 62.2% of cases, and 32.1% of outputs contained no errors, while the remaining had at least one error. ChatGPT provided 214 bibliographic references: 70.1% correct, 12.1% inaccurate, and 17.8% non-existing. Scenario variability had the greatest impact on ratings, and latent knowledge across fields showed minimal variation. Although ChatGPT provided useful responses in one-third of cases, the frequency of errors and variability underscores its inadequacy for routine diagnostic use and highlights the need for discretion as a support tool. Imprecise referencing also suggests caution as a self-learning tool. It is essential to recognize the irreplaceable role of human experts in synthesizing images, clinical data, and experience for the intricate task of histopathological diagnosis.
引用
收藏
页码:663 / 673
页数:11
相关论文
共 48 条
[1]  
Agrawal M., 2022, arXiv
[2]   Acute Pulmonary Edema After Hyperbaric Oxygen Treatment: A Case Report Written With ChatGPT Assistance [J].
Akhter, Haris M. ;
Cooper, Jeffrey S. .
CUREUS JOURNAL OF MEDICAL SCIENCE, 2023, 15 (02)
[3]  
Ali Stephen R, 2023, Lancet Digit Health, V5, pe179, DOI [10.1016/s2589-7500(23)00048-1, 10.1016/S2589-7500(23)00048-1]
[4]  
[Anonymous], 2023, Stan modeling language users guide and reference manual, version 2.33.0
[5]  
[Anonymous], 2022, OpenAI
[6]  
[Anonymous], 2023, R: a language and environment for statistical computing
[7]   Evaluating the Performance of ChatGPT in Ophthalmology [J].
Antaki, Fares ;
Touma, Samir ;
Milad, Daniel ;
El -Khoury, Jonathan ;
Duval, Renaud .
OPHTHALMOLOGY SCIENCE, 2023, 3 (04)
[8]   Unleashing the potential of AI for pathology: challenges and recommendations [J].
Asif, Amina ;
Rajpoot, Kashif ;
Graham, Simon ;
Snead, David ;
Minhas, Fayyaz ;
Rajpoot, Nasir .
JOURNAL OF PATHOLOGY, 2023, :564-577
[9]   Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum [J].
Ayers, John W. ;
Poliak, Adam ;
Dredze, Mark ;
Leas, Eric C. ;
Zhu, Zechariah ;
Kelley, Jessica B. ;
Faix, Dennis J. ;
Goodman, Aaron M. ;
Longhurst, Christopher A. ;
Hogarth, Michael ;
Smith, Davey M. .
JAMA INTERNAL MEDICINE, 2023, 183 (06) :589-596
[10]  
Benoit J. R. A., 2023, CHATGPT CLIN VIGNETT, DOI [DOI 10.1101/2023.02.04.23285478, 10.1101/2023.02.04.23285478〉]