Improving the Detection of Malware Behaviour Using Simplified Data Dependent API Call Graph

Malware stands for malicious software. It is software that is designed with a harmful intent. A malware detector is a system that attempts to identify malware using Application Programming Interface (API) call graph technique and/or other techniques. Matching the API call graph using graph matching algorithm have NP-complete problem and is slow because of computational complexity. In this study, a malware detection system based on API call graph is proposed. Each malware sample is represented as data dependent API call graph. After transforming the input sample into a simplified data dependent graph, graph matching algorithm is used to calculate similarity between the input sample and malware API call graph samples stored in a database. The graph matching algorithm is based on Longest Common Subsequence (LCS) algorithm which is used on the simplified graphs. Such strategy reduces the computation complexity by selecting paths with the same edge label in the API call graph. Experimental results on 85 samples demonstrate 98% detection rate and 0% false positive rate for the proposed malware detection system.