Measurement Practices in Large-Scale Replications: Insights From Many Labs 2

Validity of measurement is integral to the interpretability of research endeavours and any subsequent replication attempts. To assess current measurement practices and the construct validity of measures in large-scale replication studies, we conducted a systematic review of measures used in "Many Labs 2: Investigating Variation in Replicability Across Samples and Settings" (Klein et al., 2018). To evaluate the psychometric properties of the scales used in "Many Labs 2," we conducted factor and reliability analyses on the publicly available data. We report that measures in "Many Labs 2" were often short with little validity evidence reported in the original study, that measures with more validity evidence in the original study had stronger psychometric properties in the replication sample, and that translated versions of scales had lower reliability. We discuss the implications of these findings for interpreting replication results, and make recommendations to improve measurement practices in future replications.