The accuracy of Gemini, GPT-4, and GPT-4o in ECG analysis: A comparison with cardiologists and emergency medicine specialists

被引:25
作者
Gunay, Serkan [1 ]
Ozturk, Ahmet [1 ]
Yigit, Yavuz [2 ]
机构
[1] Hitit Univ, Corum Erol Olcok Educ & Res Hosp, Dept Emergency Med, Emergency Med, Corum, Turkiye
[2] Hamad Gen Hosp, Dept Emergency Med, Emergency Med, Hamad Med Corp, Doha, Qatar
关键词
Artificial intelligence; ChatGPT; GPT-4; Gemini; Electrocardiography; GPT-4o; ARTIFICIAL-INTELLIGENCE;
D O I
10.1016/j.ajem.2024.07.043
中图分类号
R4 [临床医学];
学科分类号
1002 ; 100602 ;
摘要
Introduction: GPT-4, GPT-4o and Gemini advanced, which are among the well-known large language models (LLMs), have the capability to recognize and interpret visual data. When the literature is examined, there are a very limited number of studies examining the ECG performance of GPT-4. However, there is no study in the literature examining the success of Gemini and GPT-4o in ECG evaluation. The aim of our study is to evaluate the performance of GPT-4, GPT-4o, and Gemini in ECG evaluation, assess their usability in the medical field, and compare their accuracy rates in ECG interpretation with those of cardiologists and emergency medicine specialists. Methods: The study was conducted from May 14, 2024, to June 3, 2024. The book "150 ECG Cases" served as a reference, containing two sections: daily routine ECGs and more challenging ECGs. For this study, two emergency medicine specialists selected 20 ECG cases from each section, totaling 40 cases. In the next stage, the questions were evaluated by emergency medicine specialists and cardiologists. In the subsequent phase, a diagnostic question was entered daily into GPT-4, GPT-4o, and Gemini Advanced on separate chat interfaces. In the final phase, the responses provided by cardiologists, emergency medicine specialists, GPT-4, GPT-4o, and Gemini Advanced were statistically evaluated across three categories: routine daily ECGs, more challenging ECGs, and the total number of ECGs. Results: Cardiologists outperformed GPT-4, GPT-4o, and Gemini Advanced in all three groups. Emergency medicine specialists performed better than GPT-4o in routine daily ECG questions and total ECG questions (p = 0.003 and p = 0.042, respectively). When comparing GPT-4o with Gemini Advanced and GPT-4, GPT-4o performed better in total ECG questions (p = 0.027 and p < 0.001, respectively). In routine daily ECG questions, GPT-4o also outperformed Gemini Advanced (p = 0.004). Weak agreement was observed in the responses given by GPT-4 (p < 0.001, Fleiss Kappa = 0.265) and Gemini Advanced (p < 0.001, Fleiss Kappa = 0.347), while moderate agreement was observed in the responses given by GPT-4o (p < 0.001, Fleiss Kappa = 0.514). Conclusion: While GPT-4o shows promise, especially in more challenging ECG questions, and may have potential as an assistant for ECG evaluation, its performance in routine and overall assessments still lags behind human specialists. The limited accuracy and consistency of GPT-4 and Gemini suggest that their current use in clinical ECG interpretation is risky.
引用
收藏
页码:68 / 73
页数:6
相关论文
共 50 条
[41]   Artificial Intelligence in Sports Medicine: Could GPT-4 Make Human Doctors Obsolete? [J].
Cheng, Kunming ;
Guo, Qiang ;
He, Yongbin ;
Lu, Yanqiu ;
Xie, Ruijie ;
Li, Cheng ;
Wu, Haiyang .
ANNALS OF BIOMEDICAL ENGINEERING, 2023, 51 (08) :1658-1662
[42]   GPT-4o vs. Human Candidates: Performance Analysis in the Polish Final Dentistry Examination [J].
Jaworski, Aleksander ;
Jasinski, Dawid ;
Slawinska, Barbara ;
Blecha, Zuzanna ;
Jaworski, Wojciech ;
Kruplewicz, Maja ;
Jasinska, Natalia ;
Syslo, Oliwia ;
Latkowska, Ada ;
Jung, Magdalena .
CUREUS JOURNAL OF MEDICAL SCIENCE, 2024, 16 (09)
[43]   Assessing GPT-4 multimodal performance in radiological image analysis [J].
Brin, Dana ;
Sorin, Vera ;
Barash, Yiftach ;
Konen, Eli ;
Glicksberg, Benjamin S. ;
Nadkarni, Girish N. ;
Klang, Eyal .
EUROPEAN RADIOLOGY, 2025, 35 (04) :1959-1965
[44]   Artificial Intelligence in Sports Medicine: Could GPT-4 Make Human Doctors Obsolete? [J].
Kunming Cheng ;
Qiang Guo ;
Yongbin He ;
Yanqiu Lu ;
Ruijie Xie ;
Cheng Li ;
Haiyang Wu .
Annals of Biomedical Engineering, 2023, 51 :1658-1662
[45]   Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study [J].
Meyer, Annika ;
Riese, Janik ;
Streichert, Thomas .
JMIR MEDICAL EDUCATION, 2024, 10
[46]   Assessing the accuracy of the GPT-4 model in multidisciplinary tumor board decision prediction [J].
Erdat, Efe Cem ;
Yalciner, Merih ;
Oruncu, Mehmet Berk ;
Urun, Yuksel ;
Senler, Filiz cay .
CLINICAL & TRANSLATIONAL ONCOLOGY, 2025,
[47]   Assessing the accuracy and clinical utility of GPT-4O in abnormal blood cell morphology recognition [J].
Cai, Xinjian ;
Zhan, Lili ;
Lin, Yiteng .
DIGITAL HEALTH, 2024, 10
[48]   Exploring new educational approaches in neuropathic pain: assessing accuracy and consistency of artificial intelligence responses from GPT-3.5 and GPT-4 [J].
Garcia-Rudolph, Alejandro ;
Sanchez-Pinsach, David ;
Opisso, Eloy ;
Soler, Maria Dolors .
PAIN MEDICINE, 2024, 26 (01) :48-50
[49]   Evaluating the Utility of OpenAI's GPT-4 as a Diagnostic and Management Aid in Medicine [J].
Ge, Alan ;
Pandya, Vidish ;
Ferrick, Kevin J. ;
Krumerman, Andrew .
CIRCULATION, 2023, 148
[50]   Assessing GPT-4's Performance in Delivering Medical Advice: Comparative Analysis With Human Experts [J].
Jo, Eunbeen ;
Song, Sanghoun ;
Kim, Jong -Ho ;
Lim, Subin ;
Kim, Ju Hyeon ;
Cha, Jung - Joon ;
Kim, Young -Min ;
Joo, Hyung Joon .
JMIR MEDICAL EDUCATION, 2024, 10