Integrating a Rule-Based Approach to Malware Detection with an LSTM-Based Feature Selection Technique

Technology has amplified malware activity, affecting network and users. Before being forwarded to the next host, network traffic must be dynamically analysed for malware. By exploiting network vulnerabilities, attackers gain control of the system and implement their own network rules to enable malicious traffic. Yet, another recursive acronym (YARA) rules are effective string and pattern-matching malware analysis approaches. The quality and amount of YARA rules utilized in analysis determine its effectiveness. YARA rules focus on whether to activate a rule for a suspicious sample after examining its rule condition. YARA rules rely on binary conclusion on malware analysis, which may limit its use and results. Thus, the proposed approach selects malware features using the ML-based LSTM model. Rule-based traffic analysis and long-short term memory (LSTM)-based feature selection strengthen the malware detection model in the proposed approach. By comparing performance results with and without LSTM-based feature (parameter) selection, this research assesses model integrity. Due to LSTM-based feature selection, the model achieved its best accuracy of 97%, proving its suitability for malware detection on diverse datasets belonging to different network environments. © 2023, The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd.