Learning Automated Essay Scoring Models Using Item-Response-Theory-Based Scores to Decrease Effects of Rater Biases

被引：12

作者：

Uto, Masaki ^{[1
]}

Okano, Masashi ^{[1
]}

机构：

[1] Univ Electrocommun, Grad Sch Informat & Engn, Chofu, Tokyo 1828585, Japan

来源：

IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES | 2021年 / 14卷 / 06期

基金：

日本学术振兴会;

关键词：

Data models; Training; Feature extraction; Bit error rate; Predictive models; Transformers; Task analysis; Automated essay scoring (AES); deep neural networks (DNNs); item response theory (IRT); rater bias; PSYCHOMETRIC QUALITY; PERFORMANCE;

D O I：

10.1109/TLT.2022.3145352

中图分类号：

TP39 [计算机的应用];

学科分类号：

081203 ; 0835 ;

摘要：

In automated essay scoring (AES), scores are automatically assigned to essays as an alternative to grading by humans. Traditional AES typically relies on handcrafted features, whereas recent studies have proposed AES models based on deep neural networks to obviate the need for feature engineering. Those AES models generally require training on a large dataset of graded essays. However, assigned grades in such a training dataset are known to be biased owing to effects of rater characteristics when grading is conducted by assigning a few raters in a rater set to each essay. Performance of AES models drops when such biased data are used for model training. Researchers in the fields of educational and psychological measurement have recently proposed item response theory (IRT) models that can estimate essay scores while considering effects of rater biases. This study, therefore, proposes a new method that trains AES models using IRT-based scores for dealing with rater bias within training data.

引用

页码：763 / 776

页数：14