Rating and Matching in Peer Review Systems

Peer review (e.g., review of research papers) is essential for the success of the scientific community. In peer review, the reviewers voluntarily exert costly effort in reviewing papers. Hence, it is important to design mechanisms to elicit high effort from reviewers. Exploiting the fact that the researchers interact with each other repeatedly (e.g., by submitting and reviewing papers over years), we propose a rating and matching mechanism to elicit high effort from reviewers. Our proposed mechanism overcomes two major difficulties, namely adverse selection (i.e., the unidentifiable quality of heterogeneous reviewers) and moral hazard (i.e., the unobservable effort levels from reviewers). Specifically, our proposed mechanism assigns and updates ratings for the researchers, and matches researchers' papers to reviewers with similar ratings. In this way, the mechanism identifies different types of reviewers by their ratings, and incentivizes different reviewers to exert high effort. Focusing on the matching rule, we first provide design guidelines for a general matching rule that leads the system to an equilibrium, where the reviewers' types are identified and their high efforts are elicited. Then we study in detail a baseline matching rule that assigns each researcher's paper to one of the two reviewers with the closest ratings, provide guidelines of how to choose the initial ratings, and analyze equilibrium review quality and equilibrium ratings. Finally, we extend the baseline matching rule to two classes. The first extension provides extra reward and/or punishment by adjusting the probabilities of matching each researcher's paper to its neighbors. The second extension provides extra reward and/or punishment by allowing to match each researcher's paper to reviewers other than its neighbors. We prove that it is beneficial (in the sense that the optimal equilibrium review quality is higher) to reward reviewers in the first extension, and to punish reviewers in the second extension, due to the different ways the reward and punishment are carried out. We also prove that our proposed matching rules elicit much higher effort from reviewers, compared to matching rules that mimic the current mechanisms of assigning papers.