Recursive Pyramid Network with Joint Attention for Cross-Media Retrieval

Cross-media retrieval has raised wide attention in recent years, for its flexibility in retrieving results across different media types by a query of any media type. Besides studying on the global information of the samples, some recent works focus on the regions of the samples to mine local information for better correlation learning of different media types. However, these works focus on the correlations of regions and sample, while ignoring the correlations between regions, including the significance of each region among all of them, and the supplementary information between the region and its sub-regions, similar to the sample and its regions. For addressing this problem, this paper proposes a new recursive pyramid network with joint attention (RPJA) for cross-media retrieval, which has two main contributions: (1) We repeatedly partition the sample into increasingly fine regions in a pyramid structure, and the representation of sample is generated by modeling the supplementary information, which is provided by the regions and their sub-regions recursively from the bottom to top of pyramid. (2) We propose a joint attention model connecting different media types in each pyramid level, which mines the intra-media information and inter-media correlations to guide the learning of significance of each region, further improving the performance of correlation learning. Experiments on two widely-used datasets compared with state-of-the-art methods verify the effectiveness of our proposed approach.