Decision making while rating ESL/EFL writing tasks: A descriptive framework

This article documents 3 coordinated, exploratory studies that developed a empirically a framework to describe the decisions that experienced writing assessors snake when evaluating ESL/EFL written compositions. The studies are part of ongoing research to prepare a new scoring scheme and tasks for the writing component of the Test of English as a Foreign Language (TOEFL). In Study 1 a research team of 10 experienced ESL/EFL raters developed a preliminary descriptive framework from their own think-aloud protocols while each rating (without any predefined scoring criteria) 60 TOEFL, essays at 6 different score points on 4 different essay topics. Study 2 applied the framework to verbal report data from 7 highly experienced English-mother-tongue (EMIT) composition raters while each rated 40 TOEFL essays. In Study 3 we refined the framework by analyzing think-aloud protocols from 7 of the same ESL/EFL, raters who rated compositions from 6 ESL students on 5 different writing tasks involving writing in response to reading or listening material. In each study, participants completed a questionnaire to profile their individual characteristics and relevant background variable. In addition to documenting and analyzing in detail the thinking processes of these raters, we found that both groups of racers used similar decision-making behaviors, in similar proportions of frequency, while assessing both the TOEFL essays and the new writing tasks, thus verifying the appropriateness of our descriptive framework. Raters attended snore extensively to rhetoric and ideas (compared to language) in compositions they scored high than in compositions they scored low. The ESL/EFL raters attended more extensively, though, to language than to rhetoric and ideas overall, whereas the EDIT racers balanced more evenly their attention to these mails features of tire written compositions. Most participants perceived that their previous experiences rating compositions and teaching English had influenced their criteria and their processes for rating the compositions.