Effective Image Retrieval via Multilinear Multi-Index Fusion

Multi-index fusion has demonstrated impressive performances in the retrieval task by integrating different visual representations in a unified framework. However, previous works mainly consider propagating similarities via a neighbor structure, ignoring the high-order information among different visual representations. In this paper, we propose a new multi-index fusion scheme for image retrieval. By formulating this procedure as a multilinear-based optimization problem, the complementary information hidden in different indexes can be explored more thoroughly. Specifically, we first build our multiple indexes from various visual representations. Then, a so-called index-specific functional matrix, which aims to propagate similarities, is introduced to update the original index. The functional matrices are then optimized in a unified tensor space to achieve a refinement, such that the relevant images can be pushed closer. The optimization problem can be efficiently solved by the augmented Lagrangian method with a theoretical convergence guarantee. Unlike the traditional multi-index fusion scheme, our approach embeds the multi-index subspace structure into the new indexes with sparse constraint and, thus, it has little additional memory consumption in the online query stage. Experimental evaluation on three benchmark datasets reveals that the proposed approach achieves state-of-the-art performance, that is, N-score 3.94 on UKBench, mAP 94.1 on Holiday, and 62.39 on Market-1501.