Repeatability, reproducibility, and diagnostic accuracy of a commercial large language model (ChatGPT) to perform emergency department triage using the Canadian triage and acuity scale

PurposeThe release of the ChatGPT prototype to the public in November 2022 drastically reduced the barrier to using artificial intelligence by allowing easy access to a large language model with only a simple web interface. One situation where ChatGPT could be useful is in triaging patients arriving to the emergency department. This study aimed to address the research problem: "can emergency physicians use ChatGPT to accurately triage patients using the Canadian Triage and Acuity Scale (CTAS)?".MethodsSix unique prompts were developed independently by five emergency physicians. An automated script was used to query ChatGPT with each of the 6 prompts combined with 61 validated and previously published patient vignettes. Thirty repetitions of each combination were performed for a total of 10,980 simulated triages.ResultsIn 99.6% of 10,980 queries, a CTAS score was returned. However, there was considerable variations in results. Repeatability (use of the same prompt repeatedly) was responsible for 21.0% of overall variation. Reproducibility (use of different prompts) was responsible for 4.0% of overall variation. Overall accuracy of ChatGPT to triage simulated patients was 47.5% with a 13.7% under-triage rate and a 38.7% over-triage rate. More extensively detailed text given as a prompt was associated with greater reproducibility, but minimal increase in accuracy.ConclusionsThis study suggests that the current ChatGPT large language model is not sufficient for emergency physicians to triage simulated patients using the Canadian Triage and Acuity Scale due to poor repeatability and accuracy. Medical practitioners should be aware that while ChatGPT can be a valuable tool, it may lack consistency and may frequently provide false information. ObjectifLa sortie du prototype ChatGPT au public en novembre 2022 a considerablement reduit l'obstacle a l'utilisation de l'intelligence artificielle en permettant un acces facile a un grand modele de langage avec une interface web simple. Une situation ou ChatGPT pourrait etre utile est de trier les patients qui arrivent au service d'urgence. Cette etude visait a resoudre le probleme de la recherche : << Les medecins d'urgence peuvent-ils utiliser ChatGPT pour trier avec precision les patients a l'aide de l'echelle canadienne de triage et d'acuite (ECTC) ?>>.MethodesSix invites uniques ont ete elaborees independamment par cinq urgentologues. Un script automatise a ete utilise pour interroger ChatGPT avec chacune des six invites combinees a 61 vignettes de patients validees et precedemment publiees. Trente repetitions de chaque combinaison ont ete realisees pour un total de 10980 triages simules.ResultatsDans 99.6 % des 10980 requetes, un score CTAS a ete obtenu. Cependant, il y a eu des variations considerables dans les resultats. La repetabilite (utilisation repetee de la meme invite) etait responsable de 21.0 % de la variation globale. La reproductibilite (utilisation de differentes invites) etait responsable de 4.0 % de la variation globale. La precision globale de ChatGPT pour le triage des patients simules etait de 47.5 %, avec un taux de sous-triage de 13.7 % et un taux de triage superieur de 38.7 %. Un texte plus detaille donne a titre d'invite etait associe a une plus grande reproductibilite, mais a une augmentation minimale de la precision.ConclusionsCette etude suggere que le modele actuel de ChatGPT en langage large n'est pas suffisant pour permettre aux medecins d'urgence de trier des patients simules a l'aide de l'echelle canadienne de triage et d'acuite en raison de la faible repetabilite et de la faible precision. Les medecins doivent etre conscients que, bien que ChatGPT puisse etre un outil precieux, il peut manquer de coherence et fournir frequemment de fausses informations.