Hierarchical Semantic Enhancement Network for Multimodal Fake News Detection

被引:7
作者
Zhang, Qiang [1 ]
Liu, Jiawei [1 ]
Zhang, Fanrui [1 ]
Xie, Jingyi [1 ]
Zha, Zheng-Jun [1 ]
机构
[1] Univ Sci & Technol China, Hefei, Anhui, Peoples R China
来源
PROCEEDINGS OF THE 31ST ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA, MM 2023 | 2023年
基金
国家重点研发计划; 中国国家自然科学基金;
关键词
Fake news detection; Semantic information; Multimodal; Entity; ATTENTION;
D O I
10.1145/3581783.3612423
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
The explosion of multimodal fake news content on social media has sparked widespread concern. Existing multimodal fake news detection methods have made significant contributions to the development of this field, but fail to adequately exploit the potential semantic information of images and ignore the noise embedded in news entities, which severely limits the performance of the models. In this paper, we propose a novel Hierarchical Semantic Enhancement Network (HSEN) for multimodal fake news detection by learning text-related image semantic and precise news high-order knowledge semantic information. Specifically, to complement the image semantic information, HSEN utilizes textual entities as the prompt subject vocabulary and applies reinforcement learning to discover the optimal prompt format for generating image captions specific to the corresponding textual entities, which contain multi-level cross-modal correlation information. Moreover, HSEN extracts visual and textual entities from image and text, and identifies additional visual entities from image captions to extend image semantic knowledge. Based on that, HSEN exploits an adaptive hard attention mechanism to automatically select strongly related news entities and remove irrelevant noise entities to obtain precise high-order knowledge semantic information, while generating attention mask for guiding cross-modal knowledge interaction. Extensive experiments show that our method outperforms state-of-the-art methods.
引用
收藏
页码:3424 / 3433
页数:10
相关论文
共 54 条
[31]   PRRNet: Pixel-Region relation network for face forgery detection [J].
Shang, Zhihua ;
Xie, Hongtao ;
Zha, Zhengjun ;
Yu, Lingyun ;
Li, Yan ;
Zhang, Yongdong .
PATTERN RECOGNITION, 2021, 116
[32]  
Simonyan K, 2015, Arxiv, DOI arXiv:1409.1556
[33]   Leveraging Intra and Inter Modality Relationship for Multimodal Fake News Detection [J].
Singhal, Shivangi ;
Pandey, Tanisha ;
Mrig, Saksham ;
Shah, Rajiv Ratn ;
Kumaraguru, Ponnurangam .
COMPANION PROCEEDINGS OF THE WEB CONFERENCE 2022, WWW 2022 COMPANION, 2022, :726-734
[34]  
Singhal S, 2019, 2019 IEEE FIFTH INTERNATIONAL CONFERENCE ON MULTIMEDIA BIG DATA (BIGMM 2019), P39, DOI [10.1109/BigMM.2019.00-44, 10.1109/BigMM.2019.00018]
[35]  
Song ZL, 2021, AAAI CONF ARTIF INTE, V35, P2584
[36]   Anonymous Authentication and Key Agreement Scheme Combining the Group Key for Vehicular Ad Hoc Networks [J].
Sun, Mei ;
Guo, Yuyan ;
Zhang, Dongbing ;
Jiang, MingMing .
COMPLEXITY, 2021, 2021
[37]  
Sun MZ, 2022, AAAI CONF ARTIF INTE, P4611
[38]   KAHAN: Knowledge-Aware Hierarchical Attention Network for Fake News detection on Social Media [J].
Tseng, Yu-Wun ;
Yang, Hui-Kuo ;
Wang, Wei-Yao ;
Peng, Wen-Chih .
COMPANION PROCEEDINGS OF THE WEB CONFERENCE 2022, WWW 2022 COMPANION, 2022, :868-875
[39]   JPEG Compression-aware Image Forgery Localization [J].
Wang, Menglu ;
Fu, Xueyang ;
Liu, Jiawei ;
Zha, Zheng-Jun .
PROCEEDINGS OF THE 30TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA, MM 2022, 2022, :5871-5879
[40]  
Wang Ning, 2022, ARXIV221201803