Duplicate Detection for Bayesian Network Structure Learning

We address the well-known score-based Bayesian network structure learning problem. Breadth-first branch and bound (BFBnB) has been shown to be an effective approach for solving this problem. Duplicate detection is an important component of the BFBnB algorithm. Previously, an external sorting-based technique was used for delayed duplicate detection (DDD). We propose a hashing-based technique for DDD and a bin packing algorithm for minimizing the number of external memory files and operations. We also give a structured duplicate detection approach which completely eliminates DDD. Importantly, these techniques ensure the search algorithms respect any given memory bound. Empirically, we demonstrate that structured duplicate detection is significantly faster than the previous state of the art in limited-memory settings. Our results show that the bin packing algorithm incurs some overhead, but that the overhead is offset by reducing I/O when more memory is available.