On the effectiveness of adversarial samples against ensemble learning-based windows PE malware detectors

The cybersecurity landscape is witnessing an increasing prevalence of threats and malicious programs, posing formidable challenges to conventional detection techniques. Although machine learning (ML) and deep learning (DL) have demonstrated effectiveness in malware detection, their susceptibility to adversarial attacks has led to a growing research trend. This study aims to provide a general framework that uses Reinforcement Learning and Explainable Artificial Intelligence (XAI) to generate and evaluate mutant Windows malware within the problem space. We concentrate on the three primary problems that arise while performing adversarial attacks on Windows Portable Executable malware, including format preservation, executability preservation, and maliciousness preservation. Additionally, we present an innovative approach called SHAPex to evaluate and clarify the impact of input feature predictions on malware detection predictions. This approach aims to optimize the application of results to future research efforts through three key questions pertaining to the predictive capacity of the ML/DL model. Experimental findings reveal that 100% of the selected mutation samples maintain their format integrity. Additionally, our system ensures the preservation of executable functionality in malware variants, yielding consistent and promising results. We have also encapsulated the analytical outcomes regarding the impact of input features on malware detectors' prediction decisions within a specialized framework based on three research questions, emphasizing the predictive capacity of ML/DL models.