Pilot Study on Using Large Language Models for Educational Resource Development in Japanese Radiological Technologist Exams

In this study, we explored the potential application of large language models (LLMs) to the development of educational resources for medical licensure exams in non-English-speaking contexts, focusing on the Japanese Radiological Technologist National Exam. We categorized multiple-choice questions into image-based, calculation, and textual types. We generated explanatory texts using Copilot, an LLM integrated with Microsoft Bing, and assessed their quality on a 0-4-point scale. LLMs achieved high performance for textual questions, which demonstrated their strong capability to process specialized content. However, we identified challenges in generating accurate formulas and performing calculations for calculation questions, as well as in interpreting complex medical images in image-based questions. To address these issues, we suggest using LLMs with programming functionalities for calculations and using keyword-based prompts for medical image interpretation. The findings highlight the active role of educators in managing LLM-supported learning environments, particularly by validating outputs and providing supplementary guidance to ensure accuracy. Furthermore, the rapid evolution of LLM technology necessitates continuous adaptation of utilization strategies to align with their advancing capabilities. In this study, we underscored the potential of LLMs to enhance educational practices in non-English-speaking regions, while addressing critical challenges to improve their reliability and utility.