Bias in Open Peer-Review: Evidence from the English Superior Courts

This paper explores possible biases in open peer-review using data from the English superior courts. Exploiting the random timing of on-the-job interaction between reviewers and reviewees, we find evidence that reviewers are reluctant to reverse the judgments of reviewees with whom they are about to interact, and that this effect is stronger when reviewer and reviewee share the same rank. The average bias is substantial: the proportion of reviewer affirmances is 30% points higher in the group where reviewers know they will soon work with their reviewee, relative to groups where such interaction is absent. Our results suggest reforms for the judicial listing process, and caution against recent trends in performance appraisal techniques and scientific publishing.