Ensemble Deep Manifold Similarity Learning using Hard Proxies

This paper is about deep image-similarity learning such that images of the same class have more similar deep feature representations than those belonging to different classes. For learning, prior work typically specifies loss in terms of l(2)-distancesor dot-products between deep features, despite the well-known non-Euclidean nature of deep feature spaces. Our first contribution is in specifying the N-pair loss using a manifold similarity of deep features. We introduce a new time- and memory-efficient estimation of the manifold similarities that uses a closed-form convergence solution of the Random Walk algorithm. We randomly partition the deep feature space, and express the manifold similarities via representatives of the resulting subspaces, a.k.a. proxies. Multiple random partitions of the deep feature space gives an ensemble of proxies which can be jointly used for estimating image similarity. Our second contribution is aimed at reducing overfitting by estimating hard proxies that are as close to one another as possible, but remain in their respective subspaces. We outperform the state of the art in both image retrieval and clustering on the CUB-200-2011, Cars196,and Stanford Online Products datasets with the same complexity as related ensemble methods.