An adaptive analysis framework for correlating cyber-security-related data

In recent years, due to the rise of APT attacks and the failure of traditional security facilities, organizations have to collect a large amount of cyber-security-related data and try to unveil the previously unknown attacks by analyzing them. Additionally, a report from Gartner claims, "Information security is becoming a big data analytics problem, where massive amounts of data will be correlated, analyzed and mined for meaningful patterns". Generally, the research work of big data analytics for cyber security mainly includes building big data systems, designing efficient processing algorithms and exploring specific analysis methods and applications, such as detecting DDoS attacks, identifying malicious URLs, correlating IDS alert incidents and extracting threat intelligence from certain unstructured data. Of all these work, most is the extension of previous methods in the big data context, by employing big data techniques to improve the storage capacity, accelerate the calculation or carry out correlation analysis in a much longer time window. Instead, only a few cares about the real coordination of these multi-source, heterogeneous data. In this paper, we propose an adaptive analysis framework for correlating different kinds of cyber-security-related data, such as network traffic, alert incidents and external threat intelligence. This framework can help to improve the pertinence of analysis and better discover potential threats.