When do Recommender Systems Work the Best? The Moderating Effects of Product Attributes and Consumer Reviews on Recommender Performance

We investigate the moderating effect of product attributes and consumer reviews on the efficacy of a collaborative filtering recommender system on an e-commerce site. We run a randomized field experiment on a top North American retailer's website with 184,375 users split into a recommender treated group and a control group with 37,215 unique products in the dataset. By augmenting the dataset with Amazon Mechanical Turk tagged product attributes and consumer review data from the website, we study their moderating influence on recommenders in generating conversion. We first confirm that the use of recommenders increases the baseline conversion rate by 5.9%. We find that the recommenders act as substitutes for high average review ratings with the effect of using recommenders increasing the conversion rate as much as about 1.4 additional average star ratings. Additionally, we find that the positive impacts on conversion from recommenders are greater for hedonic products compared to utilitarian products while search experience quality did not have any impact. We also find that the higher the price, the lower the positive impact of recommenders, while having lengthier product descriptions and higher review volumes increased the recommender's effectiveness. More findings are discussed in the Results. For managers, we 1) identify the products and product attributes for which the recommenders work well, 2) show how other product information sources on e-commerce sites interact with recommenders. Additionally, the insights from the results could inform novel recommender algorithm designs that are aware of strength and shortcomings. From an academic standpoint, we provide insight into the underlying mechanism behind how recommenders cause consumers to purchase.