A game-based intrusion detection mechanism to confront internal attackers

Insiders might threaten organizations' systems any time. By interacting with a system, an insider plays games with the security mechanisms employed to protect it. We apply game theory to model these interactions in an extensive form game that is being played repeatedly with an Intrusion Detection System (IDS). The outcomes of the game are quantified by first specifying players' preferences, and then, by using the von Neumann-Morgenstern utility function, to assign numbers that reflect these preferences. Examining players' best responses, the solution of the game follows by locating all the Nash Equilibria (NE). We extend the NE notion to the logit Quantal Response Equilibrium (QRE), to capture players' bounded rationality and model insider's behavior. The QRE results are more realistic, and show that the solution of the game might be significantly different than the corresponding NE solution. Thus, we determine how an insider will interact in the future, and how an IDS will react to protect the system. To easily exploit QRE results in ID, we propose the use of a detection mechanism. To present a possible implementation scheme of the detection mechanism, we give the application model and a detailed game-based detection algorithm. Categories and Subject Descriptors: C.2.0 [Computer-Communication Networks]: General Security and protection. D.4.6 [Operating Systems]: Security and Protection. (C) 2010 Elsevier Ltd. All rights reserved.