Collective Ratings for Online Labor Markets

In online labor markets, experts sell their expertise to buyers. Despite the success and the perceived promise of online labor markets, they face a serious practical challenge: providing appropriate incentives for experts to participate and exert effort to accurately (successfully) complete tasks. Personal rating schemes have been proposed to address this challenge: they provide differentiated reward/punishment to experts in order to incentivize them to cooperate (i.e. to their best to complete tasks). However, when the transactions in a market are subject to errors, the experts are wrongly punished frequently whenever personal rating schemes are deployed. This not only reduces the experts' incentives to cooperate, but also it harms the market performance such as the obtained social welfare or revenue. To mitigate the problem of wrong punishments, we develop a novel game-theoretic formalism based on collective ratings. We formalize an online labor market as a two-sided trading platform where buyers and experts interact repeatedly. The market designer's problem is to create a market policy that maximizes the market's revenue subject to the constraints imposed by the characteristics of the market and the incentives of the participants. We propose to organize such markets by dividing experts into groups for which a collective rating is created and maintained based on the buyers' aggregated feedback. We analyze how the group size and the adopted rating scheme affect the market's revenue and the social welfare of the participants in the market, and determine the optimal design of the market policy. We show that collective ratings are surprisingly more effective and more robust than personal rating for a wide variety of online labor markets.