Landmarks-based Blocking Method For Large-scale Entity Resolution

Large-scale entity resolution (ER) techniques have received tremendous attention due to the emergence of data processing within organizations and governments. The traditional ER process requires pairwise comparisons between each record when identifying records belong to the same entity, which is computationally prohibitive for large databases. With many existing indexing techniques to address this issue, it remains an open research question. We propose a landmarks-based indexing algorithm to reduce the possible pairwise comparisons of non-matches. The blocks are determined based on pre-selected records called landmarks in a multidimensional Euclidean space. The pair-wise comparisons only within these blocks reduce the search space immensely. Our method is scalable for big data entity resolution as it has O(n) insertion and query complexity.