Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study

Background: Cognitive biases in clinical decision-making significantly contribute to errors in diagnosis and suboptimal patientoutcomes. Addressing these biases presents a formidable challenge in the medical field. Objective: This study aimed to explore the role of large language models (LLMs) in mitigating these biases through the use ofthe multi-agent framework. We simulate the clinical decision-making processes through multi-agent conversation and evaluateits efficacy in improving diagnostic accuracy compared with humans. Methods: A total of 16 published and unpublished case reports where cognitive biases have resulted in misdiagnoses wereidentified from the literature. In the multi-agent framework, we leveraged GPT-4 (OpenAI) to facilitate interactions amongdifferent simulated agents to replicate clinical team dynamics. Each agent was assigned a distinct role: (1) making the finaldiagnosis after considering the discussions, (2) acting as a devil's advocate to correct confirmation and anchoring biases, (3)serving as a field expert in the required medical subspecialty, (4) facilitating discussions to mitigate premature closure bias, and(5) recording and summarizing findings. We tested varying combinations of these agents within the framework to determinewhich configuration yielded the highest rate of correct final diagnoses. Each scenario was repeated 5 times for consistency. Theaccuracy of the initial diagnoses and the final differential diagnoses were evaluated, and comparisons with human-generatedanswers were made using the Fisher exact test. Results: A total of 240 responses were evaluated (3 different multi-agent frameworks). The initial diagnosis had an accuracyof 0% (0/80). However, following multi-agent discussions, the accuracy for the top 2 differential diagnoses increased to 76%(61/80) for the best-performing multi-agent framework (Framework 4-C). This was significantly higher compared with theaccuracy achieved by human evaluators (odds ratio 3.49; P=.002). Conclusions: The multi-agent framework demonstrated an ability to re-evaluate and correct misconceptions, even in scenarioswith misleading initial investigations. In addition, the LLM-driven, multi-agent conversation framework shows promise inenhancing diagnostic accuracy in diagnostically challenging medical scenarios