Scalable malware detection system using big data and distributed machine learning approach

Computer, Internet, and Smartphone have changed our life as never before. Today, we cannot even imagine our life without these technologies. If we look around, we find everything, everywhere connected and controlled by system and software. We find amazing software and mobile applications which have become nerve of our daily life. Our dependency on this software and systems is so and so much that it is scary even to imagine, what if this system fails at any point in time. There is always a threat surrounded by various types of cyber-attacks. Every day cybercriminals are evolving their attacking strategy. Cyber-attacks using ever-more sophisticated malware are the major cause of concern for all types of users. Cyber-world has witnessed rapid changes in malware attacking strategy in the recent past. The volume, velocity, and complexity of malware are posing new challenges for malware detection systems. A scalable malware detection system with the capability to detect complex attacks is the time of need. In this paper, we have proposed a scalable malware detection system using big data and a machine learning approach. The machine learning model proposed in the system is implemented using Apache Spark which supports distributed learning. Locality-sensitive hashing is used for malware detection, which significantly reduces the malware detection time. A five-stage iterative process has been used to carry out the implementation and experimental analysis. The proposed model shown in the paper has achieved 99.8% accuracy. The proposed model has also significantly reduced the learning and malware detection time compared to models proposed by other researchers.