AUTOBIB: Automatic extraction of bibliographic information on the web

The Web has greatly facilitated access to information. However, information presented in HTML is mainly intended to be browsed by humans, and the problem of automatically extracting such information remains an important and challenging task. In this work, we focus on building a system called AUTOBIB to automate extraction of bibliographic information on the Web. We use a combination of bootstrapping, statistical, and heuristic methods to achieve a high degree of automation. To set up extraction from a new site, we only need to provide a few lines of code specifying how to download pages containing bibliographic information. We do not need to be concerned with each site's presentation format, and the system can cope with changes in the presentation format without human intervention. AUTOBIB bootstraps itself with a small seed database of structured bibliographic records. For each bibliographic Web site, we identify segments within its pages that represent bibliographic records, using state-of-the-art record-boundary discovery techniques. Next, we find matches for some of these "raw records" in the seed database using a set of heuristics. These matches serve as a training set for a parser based on the Hidden Markov Model (HMM), which is then used to parse the rest of the raw records into structured records. We have found an effective HMM structure with special states that correspond to delimiters and HTML tags in raw records. Experiments demonstrate that for our application, this HMM structure achieves high success rates without the complexity of previously proposed structures.