Defining Malicious Behavior

In this paper we propose the use of formal methods to model malicious code behavior. The paradigm shift in malware detection from conventional, signature-based static methods to evaluating dynamic system behavior is motivated by the rising number and ever-increasing sophistication of malware currently in the wild. Because of advanced polymorphic and metamorphic techniques, a purely signature-based approach is no longer sufficient for accurate malware recognition. Automating the process of behavior analysis necessitates the use of formal methods. The modeling process is built upon two cornerstones: special system call execution traces generated through dynamic analysis of suspicious code and a self-defined taxonomy of (malicious) system activities. The formal model consists of two parts: A definition of malicious behavior in the form of combinations of tasks necessary to achieve a certain malign goal and of rules for translating each task into possible patterns of system calls. Both models are realized through formal grammars. The behavior model uses the tasks as the alphabet and the grammar rules define which patterns of activities can be used to accomplish certain high-level malicious goals. The translation model on the other hand contains an attributed context-free grammar for each task. The alphabet of each grammar consists of Windows system (API) calls; the grammar rules map each task to patterns of these calls. The attributes are used to convey information contained in the parameters of the individual calls.