Balancing Efficiency and Quality in LLM-Based Entity Resolution on Structured Data

Entity Resolution (ER) is the problem of automatically determining when two or more entities refer to the same underlying entity. ER has been researched for over fifty years across multiple domains (including healthcare, e-commerce, and census data). In graph-based applications, such as deduplicating identities across (or even within) social media platforms, as well as knowledge graphs, ER can be particularly important. Traditionally, ER was a difficult problem both within Artificial Intelligence (AI) and in databases, owing to the quadratic O(n(2)) complexity of comparing n entities to each other, given one or more graphs with n total nodes. However, recent emergence of large language models (LLMs) allow us to address the challenges of ER as an AI problem, but a clear framework for applying LLMs in a cost-effective way remains an open issue. In this paper, we present such a framework and validate it through early experiments on real-world ER benchmarks. The framework is LLM-agnostic and is premised on assumptions that resemble pragmatic real-world requirements.