Military-led nation building campaigns can face many challenges as advances in certain aspects such as the economy and infrastructure may well be accompanied by setbacks in other areas such as security, making it difficult to judge overall progress. In this context, campaign assessments provide vital decision support for adapting strategy and deploying resources to best effect. The value of such assessments is enhanced when they report not only what has happened but why and when they also indicate likely future events-mirroring the three levels in Endsley's model of Situation Awareness. Our aim is to develop a process for evaluating changes to the complex organisational structure and processes used for producing assessment documents, even if it is difficult to understand the linkages between such changes and output quality. Using 2 years' worth of Australian Defence Force quarterly campaign assessments, we study the effect of adoption of a new analysis framework in their production. Using the conventions of thematic analysis and Endsley's model of Situation Awareness, we introduce the Situation Awareness Elicitation method for obtaining metrics pertaining to the value of assessments before and after introduction of the framework and perform statistical analysis on this data.
