Scalable Collaborative Filtering Recommendation Algorithm with MapReduce

Collaborative Filtering (CF) algorithm is the common solution to Recommender System (RS). With the development of network and storage technology, the amount of users and items in RS system is exclusively growing. How to increase the scalability and recommendation accuracy of CF are the main concerns in the related research. In this paper, an efficient implementation for user-based CF algorithm on MapReduce is presented. We exploit Bag of Word (BoW) method and design a hierarchical inverted index to further increase the scalability of our method. Meanwhile, a soft-assignment mechanism for the hierarchical inverted index is proposed to make up the recommendation accuracy decrease caused by the index. The Mapreduce implementations of our methods are detailed discussed and analyzed on both simulated data and real data, demonstrating that our implementation has the ability to scale to huge numbers of users and items, meanwhile ensures recommendation accuracy.