Triplet Networks Feature Masking for Sketch-Based Image Retrieval

Freehand sketches are an intuitive tool for communication and suitable for various applications. In this paper, we present an effective approach that combines triplet networks and an attention mechanism for sketch-based image retrieval (SBIR). The study conducted in this work is based on features extracted using deep convolutional neural networks (ConvNets). In order to overcome the SBIR cross-domain challenge (i.e. searching for photographs from sketch queries), we use triplet loss to train ConvNets to compute shared embedding for both sketches and images. Our main novel contribution is to combine such triplet networks with an attention mechanism. Our approach outperform previous state-of-the-art on challenging SBIR benchmarks. We achieved a recall of 41.66% (at k = 1) for the sketchy database (more than 4% improvement), a Kendal score of 42.9T(b) on the TU-Berlin SBIR benchmark (close to 5.5T(b) improvement) and a mean average precision (MAP) of 31% on Flickr15k (a category level SBIR benchmark).