Multimodal Multi-image Fake News Detection

Recent years have seen a large increase in the amount of false information that is posted online. Fake news are created and propagated in order to deceive users and manipulate opinions and subsequently have a negative impact on the society. The automatic detection of fake news is very challenging since some of those news are created in sophisticated ways containing text or images that have been deliberately modified. Combining information from different modalities can be very useful for determining which of the online articles are fake. In this paper, we propose a multimodal multi-image system that combines information from different modalities in order to detect fake news posted online. In particular, our system combines textual, visual and semantic information. For the textual representation we use the Bidirectional Encoder Representations from Transformers (BERT) to better capture the underlying semantic and contextual meaning of the text. For the visual representation we extract image tags from multiple images that the articles contain using the VGG-16 model. The semantic representation refers to the text-image similarity calculated using the cosine similarity between the title and image tags embeddings. Our experimental results on a real world dataset show that combining features from the different modalities is effective for fake news detection. In particular, our multimodal multi-image system significantly outperforms the BERT baseline by 4.19% and SpotFake by 5.39%.