Fine-grained multimodal named entity recognition with heterogeneous image-text similarity graphs

Multimodal Named Entity Recognition (MNER) leverages semantic information from multiple modalities to enhance the identification and classification of named entities in text. Effective MNER requires a thorough understanding of the intricate semantic correlations across different modalities. However, existing MNER methods often overlook fine-grained correlation information between textual and visual modalities, resulting in a loss of crucial semantic details for accurate entity recognition. We propose a novel Similarity Multimodal Reinforcement Graph (SMRG) framework for MNER to address this issue. SMRG quantifies the relevance between words and grid-level image regions to establish a nuanced heterogeneous image-text similarity graph. By leveraging the feature propagation capabilities of graph convolutional networks, SMRG captures rich semantic relationships across modalities. Moreover, SMRG employs gated aggregation modules to selectively integrate visual semantics with corresponding textual representations, thereby enhancing the expressiveness of text features for MNER. Extensive experiments on two benchmark Twitter datasets demonstrate the superiority of SMRG over state-of-the-art methods in self-domain and cross-domain scenarios.