Transformer-Enhanced Visual-Semantic Representation for Text-Image Retrieval

Text-image retrieval aims to find semantically relevant results from another modality when given an image or a sentence as a query. The key challenge is that the data representations of different modalities are inconsistent, so it is difficult to directly measure the similarity. The existing attention-based methods generally only consider the pairwise association between text and image, and ignore the semantic association between the different regions in an image which causes the learned image features or text features are insufficient at the abstract semantic level. In this paper, we propose a Transformer-Enhanced visual-semantic Representation Model (TERM) for text-image retrieval by designing Transformer-Enhanced (TE) module which play an important role in mining the context relationships between the local regions in images or between the words in sentences, therefore can provide more fine-grained clues for image and text matching. In addition, a graph structure is introduced in our method to effectively model the high-level semantics, which can accurately measure the similarity between the image and text. Experiments on Flickr30K and MSCOCO datasets demonstrate the effectiveness of our model, which achieves the state-of-the-art result for text-image retrieval. The Recall@1 on Flickr30K by our model improves image to text retrieval by 3.8% and text to image retrieval by 4.6%.