The Effects of Metacognition in Survey Research: Experimental, Cross-Sectional, and Content-Analytic Evidence

Three studies examined the role of language difficulty in public opinion questions. Guided by feelings-as-information theory (FIT; Schwarz 2011), the first study presents an experiment (N = 1,018) in which the language difficulty of public opinion questions was varied. Findings suggest that language difficulty affected metacognitive experiences, which in turn affected reports of political interest, political efficacy, don't know responding, and ideological constraint. Study 2 (N = 1,817) presents cross-sectional evidence from publicly available data that also indicates question-language difficulty influences don't know responding. Given these findings, study 3 (N = 8,090) presents a content analysis that reveals significant systematic variability in language difficulty within polling questions across 10 polling firms in 2016. Contextualizing these findings within a FIT framework, we contend that variability in language difficulty differentially and systematically affects participants' metacognitive experiences while responding to public opinion questions. Given that metacognitive experiences affect survey response, language difficulty ought to be more carefully considered when drafting opinion questions. To this end, the data presented in these studies can be used to aid question construction by providing numeric guidelines, using widely available measures that contextualize the relative difficulty of survey language. It is also recommended that items assessing metacognitive experiences be included in survey research to account for variance in this measure. At a time when polling data are ubiquitous yet polling accuracy is being called into question, it is critical to identify sources of unmeasured error within polling data.