Big-IDS: a decentralized multi agent reinforcement learning approach for distributed intrusion detection in big data networks

The growing complexity of security threats and the pervasive prevalence of cyberattacks have become more apparent in the present era, and the advent of big data, characterized by its distinctive features, has introduced layers of complexity to security tasks. Intrusion Detection Systems (IDSs) constitute a crucial line of defense, but their adaptation to the realm of big data is imperative. While traditional Machine Learning (ML)-based IDSs have been pivotal in detecting malicious patterns, they are often incapable to keep pace with the demands of expansive big data networks. This paper proposes a novel decentralized Multi-Agent Reinforcement Learning (MARL)-based IDS designed to address the specific challenges posed by big data. Our solution employs decentralized cooperative MARL, securing communicative channels throughout the detection process and concurrent data preprocessing which significantly reduces the overall processing time. Furthermore, the integration of Cloud computing and Big Data streaming techniques further facilitates real-time intrusion detection as cloud's resources allow rapid pre-process and analyse of massive data streams using powerful clusters. Likewise, Big Data streaming techniques ensure that potential intrusions are identified and addressed as they occur. Experimental results, conducted on the widely recognized NSLKDD benchmark dataset, demonstrate the superiority of our solution over other state-of-the-art approaches for big data networks, achieving an accuracy rate of 97.44%.