Testing the Reliability of Inter-Rater Reliability

被引：6

作者：

Eagan, Brendan ^{[1
]}

Brohinsky, Jais ^{[1
]}

Wang, Jingyi ^{[1
]}

Shaffer, David Williamson ^{[1
]}

机构：

[1] Univ Wisconsin, Educ Psychol, Madison, WI 53706 USA

来源：

LAK20: THE TENTH INTERNATIONAL CONFERENCE ON LEARNING ANALYTICS & KNOWLEDGE | 2020年

基金：

美国国家科学基金会;

关键词：

Interrater reliability; coding; reliability; validity; statistical analysis; LEARNING ANALYTICS; AGREEMENT; MODELS; KAPPA;

D O I：

10.1145/3375462.3375508

中图分类号：

TP301 [理论、方法];

学科分类号：

081202 ;

摘要：

Analyses of learning often rely on coded data. One important aspect of coding is establishing reliability. Previous research has shown that the common approach for establishing coding reliability is seriously flawed in that it produces unacceptably high Type I error rates. This paper focuses on testing whether or not these error rates correspond to specific reliability metrics or a larger methodological problem. Our results show that the method for establishing reliability is not metric specific, and we suggest the adoption of new practices to control Type I error rates associated with establishing coding reliability.

引用

页码：454 / 461

页数：8

共 24 条

[1] Inter-Coder Agreement for Computational Linguistics [J].

Artstein, Ron ;

Poesio, Massimo .

COMPUTATIONAL LINGUISTICS, 2008, 34 (04) :555-596

[2] What Does Methodology Mean for Learning Analytics? [J].

Bergner, Yoav ;

Gray, Geraldine ;

Lang, Charles .

JOURNAL OF LEARNING ANALYTICS, 2018, 5 (02) :1-8

[3] Metrics for Discrete Student Models: Chance Levels, Comparisons, and Use Cases [J].

Bosch, Nigel ;

Paquette, Luc .

JOURNAL OF LEARNING ANALYTICS, 2018, 5 (02) :86-104

[4]

Brennan R. L, 2001, Generalizability theory, DOI DOI 10.1007/978-1-4757-3456-0

[5] Learning As It Happens: A Decade of Analyzing and Shaping a Large-Scale Online Learning System [J].

Brinkhuis, Matthieu J. S. ;

Savi, Alexander O. ;

Hofman, Abe D. ;

Coomans, Frederik ;

van der Maas, Han L. J. ;

Maris, Gunter .

JOURNAL OF LEARNING ANALYTICS, 2018, 5 (02) :29-46

[6]

Chang J, 2009, Adv Neural Inf Process Syst, V22

[7] A COEFFICIENT OF AGREEMENT FOR NOMINAL SCALES [J].

COHEN, J .

EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1960, 20 (01) :37-46

[8] Working Together in Learning Analytics Towards the Co-Creation of Value [J].

Dollinger, Mollie ;

Liu, Danny ;

Arthars, Natasha ;

Lodge, Jason M. .

JOURNAL OF LEARNING ANALYTICS, 2019, 6 (02) :10-26

[9]

Eagan B.R., 2016, rhoR: Rho for inter rater reliability (Version 1.1.0)

[10]

Eagan Brendan, 2019, P 13 INT C COMP SUPP

← 1 2 3 →