Creativity Assessment over Time: Examining the Reliability of CAT Ratings

Theconsensual assessment technique(CAT) is a reliable and valid method to measure (product) creativity and often consideredthegold standard of creativity assessment. The reliability measure traditionally applied in CAT studies-inter-rater reliability-cannot capture time-sampling error, which is a particular relevant source of error for specific applications of the CAT. Therefore, the present study intended to investigate the test-retest reliability of CAT ratings. We asked raters (N = 61) for their creativity assessment of the same set of 90 fashion outfits at an initial rating session and a follow-up session either 2 or 4 weeks later. We found that mean product ratings-the actual focus of interest in the CAT-were highly stable over time, as evidenced by consistency and agreement ICCs clearly exceeding levels of .90. However, individual raters (partially) lacked temporal stability, indicating a drift in rater tendencies over time. Our findings support the CAT's reputation as a highly reliable measurement method, but question the temporal rating stability of the CAT's actual "measurement instrument," namely individual judges.