FRS-SIFS: fuzzy rough set session identification and feature selection in web robot detection

Nowadays, web robots are a big part of web and useful in many cases. But, there are malicious web robots that need to be detected. Web robots often conceal their navigations by sending requests with incorrect or no information. It can be quite difficult to correctly and precisely classify this kind of incomplete data, including missing values. Previous studies have used IP addresses and user agent names to overcome this challenge, but these methods are unreliable. In order to solve this challenge, this paper has presented a robust algorithm named FRS-SIFS (Fuzzy Rough Set Session Identification and Feature Selection). FRS-SIFS first identifies user sessions using fuzzy rough set clustering based on string similarity measures. It then determines important features for recognizing web users' behavioral patterns using fuzzy rough set classification. FRS-SIFS labels the sessions using a novel precise heuristic method based on four phases. Moreover, two different feature selection methods are used which include fuzzy rough set quick reduction algorithm and a novel wrapper feature selection method. Finally, the multi-objective optimization algorithm NSGA-II (non-dominated sorting genetic algorithm II) is used to select the optimal set of features. The performance of the proposed method has been evaluated on a real-world dataset by the tenfold cross-validation method. The results of the experiments have been compared with state-of-the-art methods which show the superiority of the proposed method in terms of recall, precision, and F1 measures.