The potential of Generative Pre-trained Transformer 4 (GPT-4) to analyse medical notes in three different languages: a retrospective model-evaluation study

Background Patient notes contain substantial information but are difficult for computers to analyse due to unstructured format. Large-language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4), changed our ability to process text, but we do not know how effectively they handle medical notes. We aimed to the ability of GPT-4 to answer predefined questions after reading medical notes in three different languages. Methods For this retrospective model-evaluation study, we included eight university hospitals from four countries (ie, the USA, Colombia, Singapore, and Italy). Each site submitted seven de-identified medical notes related seven separate patients to the coordinating centre between June 1, 2023, and Feb 28, 2024. Medical notes were between Feb 1, 2020, and June 1, 2023. One site provided medical notes in Spanish, one site provided notes in and the remaining six sites provided notes in English. We included admission notes, progress notes, and consultation notes. No discharge summaries were included in this study. We advised participating sites to choose medical that, at time of hospital admission, were for patients who were male or female, aged 18-65 years, had a diagnosis obesity, had a diagnosis of COVID-19, and had submitted an admission note. Adherence to these criteria was optional and participating sites randomly chose which medical notes to submit. When entering information into GPT-4, prepended each medical note with an instruction prompt and a list of 14 questions that had been chosen a Each medical note was individually given to GPT-4 in its original language and in separate sessions; the questions were always given in English. At each site, two physicians independently validated responses by GPT-4 and responded to all 14 questions. Each pair of physicians evaluated responses from GPT-4 to the seven medical notes from own site only. Physicians were not masked to responses from GPT-4 before providing their own answers, but masked to responses from the other physician. Findings We collected 56 medical notes, of which 42 (75%) were in English, seven (13%) were in Italian, seven (13%) were in Spanish. For each medical note, GPT-4 responded to 14 questions, resulting in 784 responses. 622 (79%, 95% CI 76-82) of 784 responses, both physicians agreed with GPT-4. In 82 (11%, 8-13) responses, one physician agreed with GPT-4. In the remaining 80 (10%, 8-13) responses, neither physician agreed with Both physicians agreed with GPT-4 more often for medical notes written in Spanish (86 [88%, 95% CI 79-93] 98 responses) and Italian (82 [84%, 75-90] of 98 responses) than in English (454 [77%, 74-80] of 588 responses). Interpretation The results of our model-evaluation study suggest that GPT-4 is accurate when analysing medical in three different languages. In the future, research should explore how LLMs can be integrated into clinical workflows to maximise their use in health care.