Zero-Shot Cross Modal Retrieval Method Based on Deep Supervised Learning

A novel zero-shot cross modal retrieval method based on deep supervised learning is proposed as category matching and corresponding matching are not considered in current research. Firstly, three types of image-text pairs are distinguished, including pairs from the same category that match correspondingly, pairs from the same category that do not match correspondingly, and pairs from different categories. Secondly, with the category of images and texts matched, to further realize the corresponding matching between them, two matching constraints are constructed based on different masking patterns. One is to mask samples that are of another modality and of the same category but do not match with each other, restraining the matching relations among images and texts of different categories. The other is to mask samples that are of another modality and of different categories, restraining the corresponding matching relations between images and texts of the same category. Finally, by aligning distribution structures of visual features and their corresponding semantic features in each space, the category matching and corresponding matching relations between images and texts are constrained again. In addition, to enhance the representation of text semantics, the attention mechanism is also utilized to obtain more significant sentence semantic features from word sequences. Experimental results show that on CUB dataset, the proposed method improves the image-based text retrieval and text-based image retrieval effects by 5.9% and 2.2% compared with baseline model respectively; on FLO dataset, these figures are 4.2% and 1.7% higher compared with the current best-performing methods, respectively. © 2022 Xi'an Jiaotong University. All rights reserved.