Complementing but Not Replacing: Comparing the Impacts of GPT-4 and Native-Speaker Interaction on Chinese L2 Writing Outcomes

This study explored the efficacy of large language models (LLMs), namely GPT-4, in supporting second language (L2) writing in comparison with interaction with a human language partner in the pre-writing phase. A within-subject behavioral experiment was conducted with 23 Chinese L2 learners who were exposed to three conditions: "without interaction", "interaction with GPT-4", and "interaction with a language partner". They then completed an L2 writing task. It was found that interaction with the language partner yielded significantly improved results compared with both interaction with GPT-4 and the case without interaction in terms of overall writing scores, organization, and language. Additionally, both types of interaction enhanced the participants' topic familiarity and writing confidence and reduced the task's perceived difficulty compared with the case without interaction. Interestingly, in the "interaction with GPT-4" condition, topic familiarity was positively correlated with better writing outcomes, whereas in the "interaction with a language partner" condition, perceived difficulty was positively correlated with content scores; however, content scores were negatively associated with writing confidence. This study suggests that LLMs should be used to complement and not replace human language partners in the L2 pre-writing phase.