A Language Model-Powered Simulated Patient With AutomatedFeedback for History Taking:Prospective Study

Background: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback onthe skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged aseducational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing theirrealism and potential to provide feedback. Objective: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model toprovide structured feedback on medical students'performance in history taking with a simulated patient. Methods: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot.To that end, we designed a chatbot to simulate patients'responses and provide immediate feedback on the comprehensivenessof the students'history taking. Students'interactions with the chatbot were analyzed, and feedback from the chatbot was comparedwith feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the qualityof feedback.Results: Most of the study's participants were in their third year of medical school. A total of 1894 question-answer pairs from106 conversations were included in our analysis. GPT-4's role-play and responses were medically plausible in more than 99%of cases. Interrater reliability between GPT-4 and the human rater showed "almost perfect" agreement (Cohen kappa=0.832). Lessagreement (kappa<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model's assessments wereoverly specific or diverged from human judgement. Conclusions: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medicalstudents. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overallhigh agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocatethe careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs areused in that context.