Fixing the Threshold for Effective Detection of Near Duplicate Web Documents in Web Crawling

被引：0
作者：
Narayana, V. A. [1 ]
Premchand, P. [1 ]
Govardhan, A. [1 ]
机构：
[1] CMR Coll Engn & Technol, Dept Comp Sci & Engn, Hyderabad, Andhra Pradesh, India
来源：
ADVANCED DATA MINING AND APPLICATIONS, ADMA 2010, PT I | 2010年 / 6440卷
关键词：
Fingerprint; Similarity score; Near-duplicate; Web crawling and Threshold;
D O I：
暂无
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
The drastic development of the WWW in recent times has made the concept of Web Crawling receive remarkable significance. The voluminous amounts of web documents swarming the web have posed huge challenges to web search engines making their results less relevant to the users. The presence of duplicate and near duplicate web documents in abundance has created additional overheads for the search engines critically affecting their performance and quality which have to be removed to provide users with the relevant results for their queries. In this paper, we have presented a novel and efficient approach for the detection of near duplicate web pages in web crawling where the keywords are extracted from the crawled pages and the similarity score between two pages is calculated. The documents having similarity score greater than a threshold value are considered as near duplicates. In this paper we have fixed the threshold value.
引用
收藏
页码：169 / 180
页数：12
相关论文
共 25 条
  • [1] [Anonymous], 2005, PVLDB
  • [2] [Anonymous], 2005, Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, DOI DOI 10.1145/1081870.1081956
  • [3] Bacchin M., 2003, LNCS, V2785, P161
  • [4] BALAMURUGAN S, 2008, P WORLD ACAD SCI ENG, V32
  • [5] Brin S., 1995, SIGMOD Record, V24, P398, DOI 10.1145/568271.223855
  • [6] Broder AndreiZ., 2003, WWW '03, P679, DOI DOI 10.1145/775152.775247
  • [7] Castillo C., 2005, ACM SIGIR FORUM, V39, P55
  • [8] Chakrabarti Soumen., 2002, Mining the Web: Discovering Knowledge from Hypertext Data
  • [9] Charikar M., 2002, P THIR 4 ANN ACM S T, P380
  • [10] Efficient crawling through URL ordering
    Cho, J
    Garcia-Molina, H
    Page, L
    [J]. COMPUTER NETWORKS AND ISDN SYSTEMS, 1998, 30 (1-7): : 161 - 172
123