An Empirical Analysis on Multi-turn Conversational Recommender Systems

The rise of conversational recommender systems (CRSs) brings the evolution of the recommendation paradigm, which enables users to interact with the system and achieve dynamic recommendations. As one essential branch, multi-turn CRSs, built on the user simulator paradigm, have attracted great attention due to their powerful ability to accomplish recommendations without real dialogue resources. Recent multi-turn CRS models, equipped with various delicately designed components (e.g., conversation module), achieve state-of-the-art (SOTA) performance. We, for the first time, propose a comprehensive experimental evaluation for existing SOTA multi-turn CRSs to investigate three research questions: (1) reproducibility - are the designed components beneficial to target multi-turn CRSs? (2) scenario-specific adaptability - how do these components perform in various scenarios? and (3) generality - can the effective components from the target CRS be effectively transferred to other multi-turn CRSs? To answer these questions, we design and conduct experiments under different settings, including carefully selected SOTA baselines, components of CRSs, datasets, and evaluation metrics, thus providing an experimental aspect overview of multi-turn CRSs. As a result, we derive several significant insights whereby effective guidelines are provided for future multi-turn CRS model designs across diverse scenarios.