Enhancing the Analysis of Interdisciplinary Learning Quality with GPT Models: Fine-Tuning and Knowledge-Empowered Approaches

Assessing the interdisciplinary learning quality of student learning processes is significant but complex. While some research has experimented with ChatGPT for qualitative analysis of text data through crafting prompts for tasks, the in-depth consideration of task-specific knowledge, like context and rules, is still limited. The study examined whether considering such knowledge can improve ChatGPT's labeling accuracy for interdisciplinary learning quality. The data for this research consists of 252 online posts collected during class discussions. This study utilized prompt engineering, fine-tuning, and knowledge-empowered approaches to evaluate student interdisciplinary learning and compare their accuracy. The results indicated that unmodified GPT-3.5 lacks the capability for analyzing interdisciplinary learning. Fine-tuning significantly improved the models, doubling the accuracy compared to using GPT-3.5 with prompts alone. Knowledge-empowered approaches enhanced both the prompt-based and fine-tuned models, surpassing the researchers' inter-rater reliability in assessing all dimensions of student posts. This study showcased the effectiveness of combining fine-tuning and knowledge-empowered approaches with advanced language models in assessing interdisciplinary learning, indicating the potential of applying this method for qualitative analysis in educational settings.