Distributed information retrieval systems -: The next generation

We present an open source framework for information retrieval. Our system design is based on distributed agents who crawl, parse and index data, code, and content. Our proposal will discuss the behavior patterns and attributes of several design elements and compare our system architecture with centralized search engines. The paper will describe the structure, technology and scalability of our system, and explain caching structure, routing algorithms and load balancing methods employed to improve the performance and survivability of our project. The resulting distributed framework can reduce network traffic, provide more recent results. and might even redefine information retrieval systems as well as information architectures.