Web Data Extraction from Retailers' Site using Semantic Density and Case Based Reasoning

Deep web or hidden web acts as a major source of information for many data analytical and data mining applications like product intelligence, competitive intelligence, online market intelligence etc. The web pages containing data from deep web are dynamic pages which are generated using server-side templates as a result of submission of query search form. They are not indexed to search engines and therefore cannot be retrieved using traditional keyword search. The web pages are designed to improve user experience but it makes automated processing, a unwieldy task. This makes WDE an on-going research area facing many challenges. Many solutions have been proposed over the past decades for WDE ranging from hand crafted rules to automatic template deduction and extraction. This paper explores a new framework for WDE which uses semantic density for data rich region detection and case based reasoning which helps in adapting the system to learn new templates and carry out extraction process even for unseen newly structured web pages.