More Gamification Is Not Always Better: A Case Study of Promotional Gamification in a Question Answering Website

Community Question Answering Websites (CQAs) like Stack Overflow rely on continuous user contributions to keep their services active. Nevertheless, they often undergo a sharp decline in their user participation during the holiday season, undermining their performance. To address this issue, some CQAs have developed their own special promotional gamification schemes to incentivize users to maintain their contributions throughout the holiday season. These promotional gamification schemes are often time-limited, optional, and run alongside the default gamification schemes of their websites. However, the impact of such promotional gamification schemes on user behavior remains largely unexplored in the existing literature. This paper takes the first steps toward filling this knowledge gap by conducting a large-scale empirical study of a particular promotional gamification scheme called Winter Bash (WB) on the CQA of Stack Overflow. According to our findings, promotional gamification schemes may not be the panacea they are portrayed to be. For example, in the case of WB, we find that the scheme is not effective for improving the collective engagement of all users. Only some particular user types (i.e., experienced and reputable users) are often provoked under WB. Most novice users, who comprise the majority of Stack Overflow website's user base, seem to be indifferent to such a gamification scheme. Our research also shows the importance of studying the quantity and quality of user engagement in unison to better understand the effectiveness of a gamification scheme. Previous gamification studies in the literature have focused predominantly on studying the quantity of user engagement alone. Last but not least, we conclude our paper by presenting some practical considerations for improving the design of future promotional gamification schemes in CQAs and similar platforms. © 2022 ACM.