Assessing public opinion about controversial risks: the effects of allowing an escape hatch on internal consistency and validity

Constructs designed to explain variation in outcomes are typically measured by aggregating responses to a series of scale questions; however, the choice of response categories may affect the predictive validity of the construct. This is particularly true of question scales where respondents may not have well-defined beliefs or knowledge. For example, respondents have a 50% chance of guessing the correct answer in a true or false response format. Therefore, question scales that measure objective knowledge with true or false response formats may overestimate knowledge. This study used an experimental design that randomized respondents across two groups. One group answered the standard questions in three commonly used scales, while the other group was provided an escape hatch (e.g. an 'I don't know' option). This experimental design allowed us to determine the effects of allowing an escape hatch on the internal consistency of constructs and the validity of associations between the constructs and outcome variables (i.e. opinions about human involvement in global warming and the safety of genetic modification in agriculture). Results show that while allowing an escape hatch has implications for the internal consistency of constructs measured, there was weak evidence that allowing an escape hatch affects the validity of associations between constructs and outcome variables.