Integration and Analysis of Agricultural Market Information Based on Web Mining

Agricultural big data can be used to guide agricultural production, forecast agricultural market demands, and support agricultural decisions. How to effectively extract and use the information on the Internet, which contains a large amount of agricultural information, has become a huge challenge. This paper proposes three kinds of automatic data acquisition strategies based on (focused, incremental, custom) Web crawler technologies, which are better suited to different types of agricultural websites than traditional Web crawlers. In addition to solving asynchronous processing, dynamic page rendering, distribution, and data-persistent problems encountered during data acquisition, this paper also proposes to combine the Aho-Corasick algorithm to improve the text matching efficiency. Finally, the acquired agricultural market data was visually analyzed by using key technologies of Web mining This study takes Chinese agricultural official websites, agricultural products wholesale market websites, and e-commerce websites as examples to integrate, process, visualize, and analyze the data acquired by using the three automatic data acquisition strategies proposed in this paper. (C) 2018, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved.