Research on fake news detection based on CLIP multimodal mechanism

Compared to text-only online fake news, combined graphic forms of fake news are more likely to gain people's trust. Most of these forms of fake news have been detected using multimodal feature fusion, with less attention being paid to the correlation between two modalities and the interaction within and between individual modalities. To address this problem, we propose a multimodal mechanism based on CLIP for fake news detection, referred to as CLIP-FND. First, the visual encoder and text encoder of the large-scale graphical pre-training model CLIP are used to unify and map the image and text data into the same feature space respectively. Then, the attention mechanism and the bilinear pooling method are used to fuse the text feature vector and the visual feature vector and input to the fake news detector for fake news detection. The experimental results show that the accuracy and F1 value of the model are much higher than the traditional multimodal fake news detection model on the publicly available dataset, which improves the fake news detection effect.