The big consequences of small biases: A simulation of peer review

Objective: To determine the effect of reviewer bias on grant application funding rates between a "preferred class" (PC) and a "non-preferred class" (NPC) of principal investigator. Methods: A discrete event simulation (DES) of grant review was developed which mimics the production, review, and funding determination of grants. Grants were defined to have an intrinsic quality. Three reviewers then score each grant, and assign it a value. Zero (control), one, or all reviewers may exhibit biases of varying severity against NPC investigators. Results: When total review bias exceeds 1.9% of grant score, statistically significant variation in scores between PC and NPC investigators is discernable in a pool of 2000 grant applications. When total review bias exceeds 2.8% of total grant score, statistically significant discrepancies in funding rates between PC and NPC investigators are detectable in a simulation of grant review. Conclusions: Review bias affects funding rates even when total review bias is less than half the amplitude of normal variation in an individual reviewer's score. Addressing reviewer bias will improve equity among investigators and may improve the overall quality of funded grant applications. (C) 2015 Elsevier B.V. All rights reserved.