Do Many Models Make Light Work? Evaluating Ensemble Solutions for Improved Rumor Detection

There have been many efforts to detect rumors using various machine learning (ML) models, but there is still a lack of understanding of their performance against different rumor topics and available features, resulting in a significant performance degrade against completely new and unseen (unknown) rumors. To address this issue, we investigate the relationship between ML models, features, and rumor topics to select the best rumor detection model under specific conditions using 13 different ML models. Our experiment results demonstrate that there is no clear winner among the ML models in all different rumor topics with respect to the detection performance. To overcome this problem, a possible way is to use an ensemble of ML models. Although previous work presents an improved detection of rumors using ensemble solutions (ES), their evaluation did not consider detecting unknown rumors. Further, they did not present nor evaluate the configuration of the ES to ensure that it indeed performs better than using a single ML model. Based on these observations, we propose to evaluate the use of an ES by examining their unknown rumor detection performance compared with single ML models but as well as different configurations of the ESes. Our experimental results using real-world datasets found that an ES of Random Forest, XGBoost and Multilayer perceptron overall produced the best F1 score of 0.79 for detecting unknown rumors, a significant improvement compared with a single best ML model which only achieved a 0.58 F1 score. We also showed that not all ESes are the same, with significantly degraded detection and large variations in performance when different ML models are used to construct the ES. Hence, it is infeasible to rely on any single ML model-based rumor detector. Finally, our solution also performed better than other recent detectors, such as eventAI and NileTMRG that performed similar to using a single ML model - making it a much more attractive solution to detect unknown rumors in practice.