Does change in the scoring system impact service evaluation? Evidence from Booking.com

Purpose In September 2019, Booking.com changed from the smiley-based scoring system (2.5-10) to the purely 10-point evaluation system (1-10). The smiley-based service evaluation is based on the multi-dimensional (M-D) system, whereas the purely 10-point service evaluation is based on the single-dimensional (S-D) system. This paper aims to focus on how a change in review posting policies impacts service evaluations regarding review generation and distribution. Design/methodology/approach The authors exploit the natural experiment using Booking.com when the site changed its scoring system from a multidimensional smiley-based service evaluation system to an S-D scoring system. The authors collected online reviews posted on two travel agencies (Booking.com and Priceline.com) between September 2019 and October 2020. A quasi-experimental approach, Difference-in-Differences, was used to isolate the impacts of the new scoring system from the impacts of the change in the service evaluation environment, i.e. COVID-19. Findings The change in the scoring system considerably alters review distributions by decreasing the portion of positive reviews but increasing the portion of highly positive reviews. Using the theory of emotion work (Hochschild, 1979, 2001), DID is also the reason that the former M-D smiley-based system could have underrated, highly positive reviews of services. Using the information transfer theory (Belkin, 1984), the authors reason the asymmetric transfer of information when users consume reviews from the older (M-D) system but are required to generate reviews on a newer (S-D) system. Practical implications The findings would provide online review platform management with a deeper understanding of the consequences of changes in service evaluations when the scoring system is changed. Originality/value Though the change in the scoring system would affect how customers evaluate the services of hotels, the causal impacts of switching to the new S-D scoring system have not yet been thoroughly covered by prior hospitality and service evaluation literature, which this research aspires to do.