Evolution of a Web-Scale Near Duplicate Image Detection System

被引：1

作者：

Gusev, Andrey ^{[1
]}

Xu, Jiajing ^{[1
]}

机构：

[1] Pinterest, San Francisco, CA 94107 USA

来源：

WEB CONFERENCE 2020: PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE (WWW 2020) | 2020年

关键词：

near-duplicate detection; recommendation systems; locality sensitive hashing; transfer learning; clustering;

D O I：

10.1145/3366423.3380031

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

Detecting near duplicate images is fundamental to the content ecosystem of photo sharing web applications. However, such a task is challenging when involving a web-scale image corpus containing billions of images. In this paper, we present an efficient system for detecting near duplicate images across 8 billion images. Our system consists of three stages: candidate generation, candidate selection, and clustering. We also demonstrate that this system can be used to greatly improve the quality of recommendations and search results across a number of real-world applications. In addition, we include the evolution of the system over the course of six years, bringing out experiences and lessons on how new systems are designed to accommodate organic content growth as well as the latest technology. Finally, we are releasing a human-labeled dataset of similar to 53,000 pairs of images introduced in this paper.

引用

页码：2733 / 2739

页数：7

共 33 条

[1]

[Anonymous], PROC CVPR IEEE

[2]

Chum O., 2008, BMVC, V810, DOI [10.5244/C.22.50, DOI 10.5244/C.22.50]

[3]

Chum O, 2009, PROC CVPR IEEE, P17, DOI 10.1109/CVPRW.2009.5206531

[4] Large-Scale Discovery of Spatially Related Images [J].

Chum, Ondrej ;

Matas, Jiri .

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2010, 32 (02) :371-377

[5]

Cutting D., 1990, SIGIR 90 13 INT C RE, P405, DOI DOI 10.1145/96749.98245

[6]

Eksombatchai C, 2018, WEB CONFERENCE 2018: PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE (WWW2018), P1775

[7] Scalable Object Detection using Deep Neural Networks [J].

Erhan, Dumitru ;

Szegedy, Christian ;

Toshev, Alexander ;

Anguelov, Dragomir .

2014 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2014, :2155-2162

[8] RANDOM SAMPLE CONSENSUS - A PARADIGM FOR MODEL-FITTING WITH APPLICATIONS TO IMAGE-ANALYSIS AND AUTOMATED CARTOGRAPHY [J].

FISCHLER, MA ;

BOLLES, RC .

COMMUNICATIONS OF THE ACM, 1981, 24 (06) :381-395

[9]

Gong YC, 2015, PROC CVPR IEEE, P19, DOI 10.1109/CVPR.2015.7298596

[10]

Hamilton WL, 2017, ADV NEUR IN, V30

← 1 2 3 4 →