ChatGPT Performance on the American Urological Association Self-assessment Study Program and the Potential Influence of Artificial Intelligence in Urologic Training

被引:40
作者
Deebel, Nicholas A. [1 ,2 ]
Terlecki, Ryan [1 ]
机构
[1] Wake Forest Univ, Bowman Gray Sch Med, Dept Urol, Winston Salem, NC USA
[2] Wake Forest Univ, Bowman Gray Sch Med, Dept Urol, 1 Med Ctr Blvd, Winston Salem, NC 27157 USA
关键词
D O I
10.1016/j.urology.2023.05.010
中图分类号
R5 [内科学]; R69 [泌尿科学(泌尿生殖系疾病)];
学科分类号
1002 ; 100201 ;
摘要
OBJECTIVE To assess chat generative pre-trained transformer's (ChatGPT) performance on the American Urological Association Self-Assessment Study Program (AUA SASP) and stratify performance by question stem complexity. METHODS Questions from the 2021-2022 AUA SASP program were administered to ChatGPT version 3 (ChatGPT-3). Questions were administered to the model utilizing a standardized prompt. The answer choice selected by ChatGPT was then used to answer the question stem in the AUA SASP program. ChatGPT was then prompted to assign a question stem order (first, second, third) to each question. The percentage of correctly answered questions was determined for each order level. All responses provided by ChatGPT were qualitatively assessed for appropriate rationale. RESULTS A total of 268 questions were administered to ChatGPT. ChatGPT performed better on 2021 compared to the 2022 AUA SASP question set, answering 42.3% versus 30.0% of questions correctly (P <.05). Hundred percent of answer explanations provided appropriate, relevant rationale regardless of whether the answer was correct. Further stratification included assessment by question order level. ChatGPT performed progressively better on the 2021 question set with decreasing order levels, with first-order questions reaching 53.8% (n = 14). However, differences in proportions did not reach statistical significance (P >.05). CONCLUSION ChatGPT answered many high-level questions correctly and provided a reasonable rationale for each answer choice. While ChatGPT was unable to answer numerous first-order questions, future language processing model learning may lead to the optimization of its fund of knowledge. This may lead to the utilization of artificial intelligence like ChatGPT as an educational tool for urology trainees and professors. UROLOGY 177: 29-33, 2023. (c) 2023 Elsevier Inc. All rights reserved.
引用
收藏
页码:29 / 33
页数:5
相关论文
共 11 条
  • [1] American Urological Association, 2022, Self-Assessment Study Program SASP, V2023
  • [2] The American Urological Association In-Service Examination: Performance correlates with Canadian and American specialty examinations
    Baverstock, RJ
    MacNeily, AE
    Cole, G
    [J]. JOURNAL OF UROLOGY, 2003, 170 (02) : 527 - 529
  • [3] Can ChatGPT pass the life support exams without entering the American heart association course?
    Fijaoko, Nino
    Gosak, Lucija
    Stiglic, Gregor
    Picard, Christopher T.
    Douma, Matthew John
    [J]. RESUSCITATION, 2023, 185
  • [4] Harnessing Generative Artificial Intelligence to Improve Efficiency Among Urologists: Welcome ChatGPT
    Gabrielson, Andrew T.
    Odisho, Anobel Y.
    Canes, David
    [J]. JOURNAL OF UROLOGY, 2023, 209 (05) : 827 - 829
  • [5] Gilson Aidan, 2023, JMIR Med Educ, V9, pe45312, DOI 10.2196/45312
  • [6] Hosseini Mohammad, 2023, Res Sq, DOI 10.21203/rs.3.rs-2587766/v1
  • [7] Kevin S, Microsoft teams up with OpenAI to exclusively license GPT-3 language model
  • [8] Kung T.H., 2023, PLOS Digit Health, V2, DOI [10.1101/2022.12.19.22283643, DOI 10.1371/JOURNAL.PDIG.0000198.PDIG-D-22-00371]
  • [9] Can an artificial intelligence chatbot be the author of a scholarly article?
    Lee, Ju Yoen
    [J]. JOURNAL OF EDUCATIONAL EVALUATION FOR HEALTH PROFESSIONS, 2023, 20
  • [10] Rao A, 2023, PREPRINT