Large-scale Visual Search and Similarity for E-Commerce

Visual search and similarity can aid an e-commerce platform by providing appropriate recommendations where semantic labeling and associated metadata is missing. In this work, we detail the bootstrapping of our pipeline that powers visually similar recommendations. While a common approach leverages learned representation from typical classification tasks using convolutional neural networks (CNNs), the crux of the problem are the attributes and their ontology. Our proposed approach in production for a variety of products is to supply these recommendations based on a defined taxonomy through a hierarchy that has been carefully curated while additionally scaled up through our platform's natural crowd-sourcing interface. The image representations are learned by a ResNet model architecture, trained from scratch on 3000+ classes, after applying transformations on the images using Apache Beam and Tensorflow Transforms. To scale the nearest neighbors on millions of items, we leverage quantization schemes like HNSW, IVF, and PCA. These are incrementally inferenced for new items on multiple GPUs and optimized for data throughput, and indexed in the ANN. Finally, in order to verify the appropriateness, we use an extensive human evaluation pipeline and quality control. In this work, we share all lessons learned from product design to practical applications at scale to continual deployment (training daily) from various experiments we conducted for a successful launch.