Comparison of multi-criteria decision-making methods for online controlled experiments in a launch decision-making framework

Context: User-intensive software systems such as Web and mobile applications are defined as systems that serve and interact with an increasingly large number of users. Inefficient launch decisions in user-intensive systems in domains such as social media, information retrieval and e-commerce can lead to dramatic loss in the goal metrics of these highly scalable applications, and therefore impact potentially billions of users. Objective: Due to the complexity of user-intensive systems, engineers rely heavily on A/B testing (i.e., online controlled experiments) to evaluate and measure the impact of new changes. However, little attention has been paid to improve the empirical process of making launch decisions based on the A/B testing results. In this paper, we propose a framework to address this issue. Method: We propose a Multi-Criteria Decision Making (MCDM) framework that uses A/B testing results to provide launch decisions analysis, as a complementary tool to assist decision making. The framework includes modules for 1) configuration setup, 2) criteria weighting, 3) pairwise comparison between criteria and alternatives 4) analysis of alternatives using MCDM and produces launch decisions based on the A/B testing results. Results: Experimental results from publicly available dataset that compares well-known and widely applied MDCM methods shows that a good combination of the Analysis of Alternative method (such as TOPSIS-Vector, MMOORA, and VIKOR) and Criteria Weighting method (such as Standard Deviation) in the framework can effectively assistlaunch decision making. Conclusion: We formulate the problem of launch decision making using A/B testing results and propose a MCDM based framework for it, as an imperfect first step to address this problem. The experiments suggest that MCDM methods such as TOPSIS, MMOORA and VIKOR may be effective at making launch decisions based on A/B testing results.