Assessing the Effectiveness of Student Advice Recommender Agent (SARA): the Case of Automated Personalized Feedback

Greer and Mark's (2016) paper suggested and reviewed different methods for evaluating the effectiveness of intelligent tutoring systems such as Propensity score matching. The current study aimed at assessing the effectiveness of automated personalized feedback intervention implemented via the Student Advice Recommender Agent (SARA) in a first-year biology class by means of statistical matching and by reviewing and comparing four different statistical matching methods (i.e., exact matching, nearest neighbor matching using the Mahalanobis distance, propensity score matching, and coarsened exact matching). Data from 1026 (73% female and 27% male) students who took a first-year biology course at a Western Canadian university were used. Two different measures for balance assessment of the matched data (i.e., % of balance improvement and standardized bias) were used to choose the best performing statistical matching method. Nearest neighbor matching using the Mahalanobis distance was found to be the most appropriate method for this study and results showed a statistically significant but small treatment effect for the group who received personalized feedback. Research and practical considerations were discussed and suggestions for future research are provided.