BotCapturer: Detecting botnets based on two-layered analysis with graph anomaly detection and network traffic clustering

Botnets have become one of the most serious threats on the Internet. On the platform of botnets, attackers conduct series of malicious activities such as distributed denial-of-service (DDoS) or virtual currencies mining. Network traffic has been widely used as the data source for the detection of botnets. However, there are two main issues on the detection of botnets with network traffic. First, many traditional filtering methods such as whitelisting are not able to process the very large amount of traffic data in real-time due to their limited computational capability. Second, many existing detection methods, based on network traffic clustering, result in high false positive rates. In this work, we are motivated to resolve the above two issues by proposing a lightweight botnet detection system called BotCapturer, based on two-layered analysis with anomaly detection in graph and network communication traffic clustering. First, we identify anomalous nodes that correspond to C&C (Control and Command) servers with anomaly scores in a graph abstracted from the network traffic. Second, we take advantage of clustering algorithms to check whether the nodes interacting with an anomalous node share similar communication pattern. In order to minimize irrelevant traffic, we propose a traffic reduction method to reduce more than 85% background traffic. The reduction is conducted by filtering the packets that are unrelated to the hosts like C&C server. We collect a very big dataset by simulating five different botnets and mixing the collected traffic with background traffic obtained from ISP. Extensive experiments are conducted and evaluation results based on our own dataset show that BotCapturer reduces more than 85% input raw packet traces and achieves a high detection rate (100%) with a low false positive rate (0.01%), demonstrating that it is very effective and efficient in detecting latest botnets. © 2018 Totem Publisher, Inc. All rights reserved.