IRPDP_HT2: a scalable data pre-processing method in web usage mining using Hadoop MapReduce

Data preparation is a vital step in the web usage mining process since it provides structured data for the subsequent stages. Hence, it is necessary to convert raw server logs into user sessions to generate structured data for pattern discovery phase. In recent decade, popular websites' server log production has risen to many terabytes to petabytes each day. As a result, server logs possess big data issues such as storage and processing. This study focuses on initial phases of web usage mining process such as data cleaning, user identification, and session identification. These phases are classified as data-intensive processes and deemed-computation intensive. In the last decade, MapReduce emerges as one of the best parallel programming frameworks for data-intensive applications. An efficient MapReduce-based data pre-processing algorithm, i.e. IRPDP_HT2, is proposed in this study. Previous parallel data pre-processing algorithms either include partial phases or lack with efficient robot detection approaches. IRPDP_HT2 algorithm uses a variety of efficient heuristics in all three phases of data pre-processing to identify both ethical and unethical robots. The suggested IRPDP_HT2 approach is found to be effective and scalable for larger datasets after various experiments on a cluster of nodes. The effectiveness of suggested heuristics is also examined during session identification phase. Three variants of IRPDP_HT2 such as PDP_HT2, IPDP_HT2, and RPDP_HT2 are also developed and tested. Impact of robots' requests and internal dummy connections' requests on session count by IRPDP_HT2 algorithm is 45.81% which is more than in PDP_HT2, IPDP_HT2, and RPDP_HT2 algorithms. Further speed-up and size-up are also analysed to demonstrate scalability of algorithm. In the presence of larger datasets, the algorithm's running time falls, while the number of data nodes grows. The size-up of IRPDP_HT2 demonstrates that even after doubling the input data, the algorithm's running time does not grow in that ratio for the fixed number of data nodes.