Evaluating the Cognitive Reflection Test as a Measure of Intuition/Reflection, Numeracy, and Insight Problem Solving, and the Implications for Understanding Real-World Judgments and Beliefs

The cognitive reflection test (CRT) has increasingly dominated theorizing about individual differences in intuitive/reflective thinking propensities, and it is associated with many real-world beliefs and judgments, such as religiosity, paranormal beliefs, and moral judgments. The CRT triggers common incorrect responses that come to mind easily, and it is frequently assumed that recognizing this error is tantamount to solving the problems. As a result, incorrect answers on the CRT purportedly indicate an intuitive thought process, whereas correct answers purportedly indicate a reflective thought process. It has also been argued that the CRT problems are fundamentally different from insight problems because insight problems often cause people to sit lost in thought, unable to identify a solution until they correctly reframe it. The present research tested these assumptions and found that a substantial proportion of people have difficulty solving the CRT problems even when the "intuitive" response is unavailable to them, the correct answer is among four multiple-choice options, and they take time to reflect. Associations between the CRT and beliefs (religiosity, paranormal beliefs, moral judgments, etc.) remained even under conditions in which CRT errors appeared to result from more reflective thought than correct responses. Furthermore, multidimensional item response theory models indicated that the CRT loaded onto numeracy and insight problem solving ability factors rather than its own unique factor. Regression analyses also indicated that numeracy and insight may account for many associations between the CRT and real-world beliefs. Broader implications for dual-process theories of reasoning and judgment are discussed.