Tree and Graph Based Two-Stages Routing for Approximate Nearest Neighbor Search

被引:0
作者
Li, Jiannan [1 ]
Zhang, Zhenyu [1 ]
Wang, Xiaoling [1 ]
Li, Haoyang [2 ]
机构
[1] East China Normal Univ, Shanghai, Peoples R China
[2] Montverde Acad Shanghai, Shanghai, Peoples R China
来源
WEB AND BIG DATA, APWEB-WAIM 2024, PT III | 2024年 / 14963卷
基金
国家重点研发计划;
关键词
Approximate nearest neighbor search; Reinforcement learning; Graph Index; Tree Index; Hybrid Index;
D O I
10.1007/978-981-97-7238-4_24
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
With the expansion of extensive datasets of high-dimensional vectors, the approximate nearest neighbor search (ANNS) has become increasingly significant in data mining. Among the prevailing ANNS methods, graph-based methods achieve high recall at the cost of excessive memory usage, whereas tree-based methods are memory-efficient but suffer from low recall and increased search complexity. In this paper, we propose a hybrid index based on Ball tree and Hierarchical Navigable Small World(HNSW). Ball-HNSW employs a two-stages hybrid framework to accelerate the identification of approximate nearest neighbors. For datasets with different distributions, we utilize reinforcement learning to optimize parameter combinations by simulating decisions during the search process, thereby improving search results. Experiments confirm that Ball-HNSW outperforms other state-of-the-art methods on multiple datasets.
引用
收藏
页码:376 / 390
页数:15
相关论文
共 50 条
  • [41] Binary Hashing for Approximate Nearest Neighbor Search on Big Data: A Survey
    Cao, Yuan
    Qi, Heng
    Zhou, Wenrui
    Kato, Jien
    Li, Keqiu
    Liu, Xiulong
    Gui, Jie
    IEEE ACCESS, 2018, 6 : 2039 - 2054
  • [42] Learning Adaptive Hypersphere: Boosting Efficiency on Approximate Nearest Neighbor Search
    Ai, Liefu
    Jiang, Changyu
    IEEE SIGNAL PROCESSING LETTERS, 2024, 31 : 2190 - 2194
  • [43] Efficient Approximate Nearest Neighbor Search by Optimized Residual Vector Quantization
    Ai, Liefu
    Yu, Junqing
    Guan, Tao
    He, Yunfeng
    2014 12TH INTERNATIONAL WORKSHOP ON CONTENT-BASED MULTIMEDIA INDEXING (CBMI), 2014,
  • [44] Learning-based query optimization for multi-probe approximate nearest neighbor search
    Zhang, Pengcheng
    Yao, Bin
    Gao, Chao
    Wu, Bin
    He, Xiao
    Li, Feifei
    Lu, Yuanfei
    Zhan, Chaoqun
    Tang, Feilong
    VLDB JOURNAL, 2023, 32 (03) : 623 - 645
  • [45] Learning-based query optimization for multi-probe approximate nearest neighbor search
    Pengcheng Zhang
    Bin Yao
    Chao Gao
    Bin Wu
    Xiao He
    Feifei Li
    Yuanfei Lu
    Chaoqun Zhan
    Feilong Tang
    The VLDB Journal, 2023, 32 : 623 - 645
  • [46] Multiple Query-independent Values based Asymmetric Ranking for Approximate Nearest Neighbor Search
    Cao, Yuan
    Qi, Heng
    Li, Keqiu
    Stojmenovic, Milos
    2016 IEEE TRUSTCOM/BIGDATASE/ISPA, 2016, : 1628 - 1635
  • [47] Massive parallelization of approximate nearest neighbor search on KD-tree for high-dimensional image descriptor matching
    Hu, Linjia
    Nooshabadi, Saeid
    JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION, 2017, 44 : 106 - 115
  • [48] TreeCANN - k-d Tree Coherence Approximate Nearest Neighbor Algorithm
    Olonetsky, Igor
    Avidan, Shai
    COMPUTER VISION - ECCV 2012, PT IV, 2012, 7575 : 602 - 615
  • [49] AnnexML: Approximate Nearest Neighbor Search for Extreme Multi-label Classification
    Tagami, Yukihiro
    KDD'17: PROCEEDINGS OF THE 23RD ACM SIGKDD INTERNATIONAL CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING, 2017, : 455 - 464
  • [50] HDIdx: High-dimensional indexing for efficient approximate nearest neighbor search
    Wan, Ji
    Tang, Sheng
    Zhang, Yongdong
    Li, Jintao
    Wu, Pengcheng
    Hoi, Steven C. H.
    NEUROCOMPUTING, 2017, 237 : 401 - 404