Assessing ChatGPT's ability to pass the FRCS orthopaedic part A exam: A critical analysis

被引：20

作者：

Saad, Ahmed ^{[1
]}

Iyengar, Karthikeyan P. ^{[2
]}

Kurisunkal, Vineet ^{[3
]}

Botchu, Rajesh ^{[4
,5
]}

机构：

[1] Royal Orthoped Hosp, Dept Orthoped, Birmingham, England

[2] Southport & Ormskirk Hosp NHS Trust, Dept Orthoped, Southport, England

[3] Royal Orthoped Hosp, Dept Orthoped Oncol, Birmingham, England

[4] Royal Orthoped Hosp, Dept Musculoskeletal Radiol, Birmingham, England

[5] Royal Orthoped Hosp, Dept Musculoskeletal Radiol, Bristol Rd South Northfield, Birmingham, England

来源：

SURGEON-JOURNAL OF THE ROYAL COLLEGES OF SURGEONS OF EDINBURGH AND IRELAND | 2023年 / 21卷 / 05期

关键词：

Artificial intelligence (AI); FRCS orthopaedics; Medical exams; ChatGPT;

D O I：

10.1016/j.surge.2023.07.001

中图分类号：

R61 [外科手术学];

学科分类号：

摘要：

AI technology has made significant advancements in recent years, with the notable development of ChatGPT in November 2022. Users have observed evidence of deductive reasoning, logical thinking, and coherent thought in ChatGPT's responses. This study aimed to determine if ChatGPT has the capability to pass the Orthopaedic Fellow of the Royal College of Surgeons (FRCS Orth) Part A exam.Methods: To assess ChatGPT4's ability to pass the Orthopaedic FRCS Orth Part A exam, a study was conducted using 240 mock FRCS Orth Part A questions. The study evaluated the accuracy of ChatGPT's answers and the response time for each question. Descriptive statistics were employed to analyse the chatbot's performance.Results: The evaluation revealed that ChatGPT4 achieved an overall score of 67.5% on Part A of the exam. However, ChatGPT4 did not meet the overall pass mark required for the FRCS Orth Part A exam.Conclusion: This study demonstrates that ChatGPT was unable to pass the FRCS Orthopaedic examination. Several factors contributed to this outcome, including the lack of critical or high-order thinking abilities, limited clinical expertise, and the inability to meet the rigorous requirements of the exam.(c) 2023 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved.

引用

页码：263 / 266

页数：4