Performance of ChatGPT on the National Korean Occupational Therapy Licensing Examination

被引：5
作者：
Lee, Si-An [1 ]
Heo, Seoyoon [2 ]
Park, Jin-Hyuck [3 ,4 ]
机构：
[1] Soonchunhyang Univ, Grad Sch, Dept ICT Convergence, Asan, South Korea
[2] Kyungbok Univ, Dept Occupat Therapy, Namyangju, South Korea
[3] Soonchunhyang Univ, Coll Med Sci, Dept Occupat Therapy, Asan, South Korea
[4] Soonchunhyang Univ, Coll Med Sci, Dept Occupat Therapy, Room 1401,22 Soonchunhyang Ro, Asan 31538, Chungcheongnam, South Korea
来源：
DIGITAL HEALTH | 2024年 / 10卷
基金：
新加坡国家研究基金会;
关键词：
ChatGPT; large language models; occupational therapy; licensing examination; artificial intelligence;
D O I：
10.1177/20552076241236635
中图分类号：
R19 [保健组织与事业（卫生事业管理）];
学科分类号：
摘要：
Background ChatGPT is an artificial intelligence-based large language model (LLM). ChatGPT has been widely applied in medicine, but its application in occupational therapy has been lacking.Objective This study examined the accuracy of ChatGPT on the National Korean Occupational Therapy Licensing Examination (NKOTLE) and investigated its potential for application in the field of occupational therapy.Methods ChatGPT 3.5 was used during the five years of the NKOTLE with Korean prompts. Multiple choice questions were entered manually by three dependent encoders, and scored according to the number of correct answers.Results During the most recent five years, ChatGPT did not achieve a passing score of 60% accuracy and exhibited interrater agreement of 0.6 or higher.Conclusion ChatGPT could not pass the NKOTLE but demonstrated a high level of agreement between raters. Even though the potential of ChatGPT to pass the NKOTLE is currently inadequate, it performed very close to the passing level even with only Korean prompts.
引用
收藏
页数：8
相关论文
共 24 条
  • [1] ChatGPT and the Future of Medical Writing
    Biswas, Som
    [J]. RADIOLOGY, 2023, 307 (02)
  • [2] Boop C., 2020, Am J Occup Ther, V74, P84
  • [3] Brown TB, 2020, ADV NEUR IN, V33
  • [4] Castelvecchi Davide, 2022, Nature, DOI 10.1038/d41586-022-04383-z
  • [5] Fang CC, 2023, medRxiv, DOI [10.1101/2023.05.03.23289443, 10.1101/2023.05.03.23289443, DOI 10.1101/2023.05.03.23289443, 10.1101/2023.05.03.23289443v1]
  • [6] Hacker P, 2023, Arxiv, DOI [arXiv:2302.02337, 10.48550/arXiv.2302.02337, 10.48550/ARXIV.2302.02337]
  • [7] Hu J., 2022, P 39 INT C MACHINE L, P4411
  • [8] Are ChatGPT's knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study
    Huh, Sun
    [J]. JOURNAL OF EDUCATIONAL EVALUATION FOR HEALTH PROFESSIONS, 2023, 20
  • [9] Kasai J, 2023, Arxiv, DOI [arXiv:2303.18027, DOI 10.48550/ARXIV.2303.18027]
  • [10] kuksiwon.or.kr, Performance Data
123