Log Analysis of Cyber Security Training Exercises

Cyber security is a pervasive issue that impacts public and private organizations. While several published accounts describe the task demands of cyber security analysts, it is only recently that research has begun to investigate the cognitive and performance factors that distinguish novice from expert cyber security analysts. Research in this area is motivated by the need to understand how to better structure the education and training of cyber security professionals, a desire to identify selection factors that are predictive of professional success in cyber security and questions related to the development of software tools to augment human performance of cyber security tasks. However, a common hurdle faced by researchers involves gaining access to cyber security professionals for data collection activities, whether controlled experiments or semi-naturalistic observations. An often readily available and potentially valuable source of data may be found in the records generated through cyber security training exercises. These events frequently entail semi-realistic challenges that may be modeled on real-world occurrences, and occur outside normal operational settings, freeing participants from the sensitivities regarding information disclosure within operational environments. This paper describes an infrastructure tailored for the collection of human performance data within the context of cyber security training exercises. Techniques are described for mining the resulting data logs for relevant human performance variables. The results provide insights that go beyond current descriptive accounts of the cognitive processes and demands associated with cyber security job performance, providing quantitative characterizations of the activities undertaken in solving problems within this domain. (C) 2015 The Authors. Published by Elsevier B.V.