Reviewer, essay, and reviewing-process characteristics that predict errors in web-based peer review

被引:7
作者
Xiong, Yao [1 ]
Schunn, Christian D. [2 ]
机构
[1] Pearson Assessments, San Antonio, TX USA
[2] Univ Pittsburgh, Pittsburgh, PA 15260 USA
关键词
Peer review; Review error; Essay; Reviewer; Reviewing process characteristics that predict errors in web-based peer review; METAANALYSIS COMPARING PEER; WRITING REVISION; FEEDBACK; QUALITY; EXPERT; IMPACT; SELF; PERCEPTIONS; RELIABILITY; VALIDITY;
D O I
10.1016/j.compedu.2021.104146
中图分类号
TP39 [计算机的应用];
学科分类号
081203 ; 0835 ;
摘要
Accuracy of peer review continues to be a concern for instructors in implementing computer-supported peer review in their instructional practices. A large body of literature has descriptively documented overall levels of reliability and validity of peer review and which factors across different peer review implementations impact overall reliability and validity of peer review (e.g., use of rubrics, education level, training). However, few studies have examined what factors within a peer review implementation contribute to review accuracy of individual reviews and knowledge about these factors could shape new interventions to avoid or remediate errors in particular reviews. In the current study, we tested a three-level framework (reviewer, essay, and reviewing process) for predicting the location of peer review errors. Further, we examined what factors within each level are predictive of two different types of review errors: severity and leniency. Leveraging a large dataset from an Advanced Placement English and Composite course implementing a common assignment with web-based peer review across 10 high schools, we found support for all levels in the framework and the importance of separating severity and leniency errors: review comment length predicted both severe and lenient errors but in opposite directions: longer comments are more likely to be associated with severe errors and less likely to be associated with lenient errors; review disagreement, reviewer ability and average sentence length of comments predicted severe errors; and essay quality predicted lenient errors. Implications for the development of new web-based tools for supporting peer-review are discussed.
引用
收藏
页数:18
相关论文
共 55 条
[1]  
AERA APA & NCME, 2014, STAND ED PSYCH TEST
[2]   The impact of peer solution quality on peer-feedback provision on geometry proofs: Evidence from eye-movement analysis [J].
Alqassab, Maryam ;
Strijbos, Jan-Willem ;
Ufer, Stefan .
LEARNING AND INSTRUCTION, 2018, 58 :182-192
[3]  
[Anonymous], 2013, P 6 INT C ED DAT MIN
[4]   Fitting Linear Mixed-Effects Models Using lme4 [J].
Bates, Douglas ;
Maechler, Martin ;
Bolker, Benjamin M. ;
Walker, Steven C. .
JOURNAL OF STATISTICAL SOFTWARE, 2015, 67 (01) :1-48
[5]   Syntactic complexity as a predictor of adolescent writing quality: Which measures? Which genre? [J].
Beers, Scott F. ;
Nagy, William E. .
READING AND WRITING, 2009, 22 (02) :185-200
[6]   CALCULATION OF POLYCHOTOMOUS LOGISTIC-REGRESSION PARAMETERS USING INDIVIDUALIZED REGRESSIONS [J].
BEGG, CB ;
GRAY, R .
BIOMETRIKA, 1984, 71 (01) :11-18
[7]   Reliability and validity of Web-based portfolio peer assessment: A case study for a senior high school's students taking computer course [J].
Chang, Chi-Cheng ;
Tseng, Kuo-Hung ;
Chou, Pao-Nan ;
Chen, Yi-Hui .
COMPUTERS & EDUCATION, 2011, 57 (01) :1306-1316
[8]   Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system [J].
Cho, Kwangsu ;
Schunn, Christian D. .
COMPUTERS & EDUCATION, 2007, 48 (03) :409-426
[9]   Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives [J].
Cho, Kwangsu ;
Schunn, Christian D. ;
Wilson, Roy W. .
JOURNAL OF EDUCATIONAL PSYCHOLOGY, 2006, 98 (04) :891-901
[10]   Student revision with peer and expert reviewing [J].
Cho, Kwangsu ;
MacArthur, Charles .
LEARNING AND INSTRUCTION, 2010, 20 (04) :328-338