Distributed Web spider based on intelligent agent

被引：0

作者：

Dong, MK ^{[1
]}

Shi, ZZ ^{[1
]}

机构：

[1] Chinese Acad Sci, Inst Comp Technol, Beijing 100080, Peoples R China

来源：

WORLD WIDE WEB TECHNOLOGIES IN CHINA: RESEARCH, DEVELOPMENT, AND APPLICATIONS | 2002年

关键词：

intelligent agent; multi-agent; agent model; distributed web spider; search engine;

D O I：

暂无

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

Web information collection is a main approach to obtain information. Also it is the important component of web applications, such as search engine, information service, web data mining and so on. Based on agent theory and technology, this paper presents a distributed web spider model for web information collection. Through modeling the special problem domain and analyzing the instantial models, an agent-oriented modeling method for distributed system is formed. The basic framework of Multi-Agent Environment (MAGE) is presented. The strategies of construct, instantial model and work mechanism are also discussed. The model of distributed web spider features knowledge-based, distributed, configurable, modular and integrated. Finally, we implement the distributed web spider system under the framework of MAGE and present experiments and analysis.

引用

页码：148 / 162

页数：15

共 15 条

[1]

[Anonymous], TOOLS

[2] The anatomy of a large-scale hypertextual Web search engine [J].

Brin, S ;

Page, L .

COMPUTER NETWORKS AND ISDN SYSTEMS, 1998, 30 (1-7) :107-117

[3]

BURNER M, 1997, WEB TECHNIQUES MAGAZ, V2

[4]

Chen HC, 1998, DECIS SUPPORT SYST, V23, P41, DOI 10.1016/S0167-9236(98)00035-9

[5]

Edwards Jenny, 2001, Proceedings of the Tenth Conference on World Wide Web, P106, DOI [DOI 10.1145/371920.371960, 10.1145/371920.371960]

[6]

Eichmann D., 1994, P 1 INT WORLD WID WE, P113

[7] Mercator: A scalable, extensible Web crawler [J].

Heydon A. ;

Najork M. .

World Wide Web, 1999, 2 (4) :219-229

[8]

HIRAI J, 2000, P 9 INT WORLD WID WE, P277

[9]

Jennings N. R., 1998, AGENT TECHNOLOGY FDN, P3, DOI [DOI 10.1007/978-3-662-03678-5_1, 10.1007/978-3-662-03678-5_1]

[10] SPHINX: a framework for creating personal, site-specific Web crawlers [J].

Miller, RC ;

Bharat, K .

COMPUTER NETWORKS AND ISDN SYSTEMS, 1998, 30 (1-7) :119-130

← 1 2 →