Performance of ChatGPT in Ophthalmic Registration and ClinicalDiagnosis:Cross-Sectional Study

Background: Artificial intelligence (AI) chatbots such as ChatGPT are expected to impact vision health care significantly.Their potential to optimize the consultation process and diagnostic capabilities across range of ophthalmic subspecialties haveyet to be fully explored.Objective: This study aims to investigate the performance of AI chatbots in recommending ophthalmic outpatient registrationand diagnosing eye diseases within clinical case profiles.Methods: This cross-sectional study used clinical cases from Chinese Standardized Resident Training-Ophthalmology (2ndEdition). For each case, 2 profiles were created: patient with history (Hx) and patient with history and examination (Hx+Ex).These profiles served as independent queries for GPT-3.5 and GPT-4.0 (accessed from March 5 to 18, 2024). Similarly, 3ophthalmic residents were posed the same profiles in a questionnaire format. The accuracy of recommending ophthalmicsubspecialty registration was primarily evaluated using Hx profiles. The accuracy of the top-ranked diagnosis and the accuracyof the diagnosis within the top 3 suggestions (do-not-miss diagnosis) were assessed using Hx+Ex profiles. The gold standard forjudgment was the published, official diagnosis. Characteristics of incorrect diagnoses by ChatGPT were also analyzed.Results: A total of 208 clinical profiles from 12 ophthalmic subspecialties were analyzed (104 Hx and 104 Hx+Ex profiles).For Hx profiles, GPT-3.5, GPT-4.0, and residents showed comparable accuracy in registration suggestions (66/104, 63.5%;81/104, 77.9%; and 72/104, 69.2%, respectively; P=.07), with ocular trauma, retinal diseases, and strabismus and amblyopiaachieving the top 3 accuracies. For Hx+Ex profiles, both GPT-4.0 and residents demonstrated higher diagnostic accuracy thanGPT-3.5 (62/104, 59.6% and 63/104, 60.6% vs 41/104, 39.4%; P=.003 and P=.001, respectively). Accuracy for do-not-missdiagnoses also improved (79/104, 76% and 68/104, 65.4% vs 51/104, 49%; P<.001 and P=.02, respectively). The highest diagnosticaccuracies were observed in glaucoma; lens diseases; and eyelid, lacrimal, and orbital diseases. GPT-4.0 recorded fewer incorrecttop-3 diagnoses (25/42, 60% vs 53/63, 84%; P=.005) and more partially correct diagnoses (21/42, 50% vs 7/63 11%; P<.001)than GPT-3.5, while GPT-3.5 had more completely incorrect (27/63, 43% vs 7/42, 17%; P=.005) and less precise diagnoses(22/63, 35% vs 5/42, 12%; P=.009).Conclusions: GPT-3.5 and GPT-4.0 showed intermediate performance in recommending ophthalmic subspecialties for registration.While GPT-3.5 underperformed, GPT-4.0 approached and numerically surpassed residents in differential diagnosis. AI chatbotsshow promise in facilitating ophthalmic patient registration. However, their integration into diagnostic decision-making requiresmore validation