Quantifying Bias and Variance of System Rankings

被引：6

作者：

Cormack, Gordon V. ^{[1
]}

Grossman, Maura R. ^{[1
]}

机构：

[1] Univ Waterloo, Waterloo, ON, Canada

来源：

PROCEEDINGS OF THE 42ND INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '19) | 2019年

关键词：

D O I：

10.1145/3331184.3331356

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

When used to assess the accuracy of system rankings, Kendall's tau and other rank correlation measures conflate bias and variance as sources of error. We derive from t a distance between rankings in Euclidean space, from which we can determine the magnitude of bias, variance, and error. Using bootstrap estimation, we show that shallow pooling has substantially higher bias and insubstantially lower variance than probability-proportional-to-size sampling, coupled with the recently released dynAP estimator.

引用

页码：1089 / 1092

页数：4