Harnessing the power of ChatGPT to develop effective MCQ-based clinical pharmacy exams

AI has the potential to revolutionize learning and assessments. This research explored ChatGPT's potential in creating MCQ-based clinical pharmacy exam papers at the Pharm-D level and compared their quality to human-made exams. In this study, three different MCQ-based papers (2 by ChatGPT; AIeasy & AIhard, and 1 by instructor; HUMAN) were set to have 10 MCQs each. We asked ChatGPT to set one exam with a low level of difficulty (AIeasy) and a second exam with specific instructions to achieve a more difficult exam (AIhard). Students attempted these exams as part of their regular assessment and rated them from 1 to 5 for various parameters such as critical thinking involved, difficulty level, and overall experience. Our study shows that students obtained higher marks for AIeasy (7.67 +/- 3.92) than for AIhard (7.06 +/- 1.32) and HUMAN (5.02 +/- 1.70), making the HUMAN exam the most difficult one. Students rated the AIhard and HUMAN exams higher for critical thinking than for AIeasy. The students rated highest for AIeasy for overall experience. Interestingly, most of the students (n = 52, 83%) could not correctly identify the exam set by the instructor. Therefore, with clear instructions, ChatGPT can create content-relevant, good-quality exam papers with varying difficulty levels. This is especially useful for students who need to self-study and be ready for several exams to gauge their knowledge.