Composite Reliability of Multilevel Data: It's About Observed Scores and Construct Meanings

This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel modeling. In particular, it discusses the contributions and limitations of the various level-specific reliability indices proposed by Geldhof, Preacher, and Zyphur (2014), denoted as (omega) over tilde (b) and (omega) over tilde (w) (and also (alpha) over tilde (b) and (alpha) over tilde (w)). One major limitation of those indices is that they are quantities for latent, unobserved level-specific composite scores, and are not suitable for observed composites at different levels. As illustrated using simulated data in this article, (omega) over tilde can drastically overestimate the true reliability of between-level composite scores (i.e., observed cluster means). Another limitation is that the development of those indices did not consider the recent conceptual development on construct meanings in multilevel modeling (Stapleton & Johnson, 2019; Stapleton, Yang, & Hancock, 2016). To address the second limitation, this article defines reliability indices (omega(2l), omega(b), omega(w), alpha(2l), alpha(b), alpha(w)) for three types of multilevel observed composite scores measuring various multilevel constructs: individual, configural, shared, and within-cluster. The article also shows how researchers can obtain sample point and interval estimates using the derived formulas and the provided R and Mplus code. In addition, a large-scale national data set was used to illustrate the proposed methods for estimating reliability for the three types of multilevel composite scores, and practical recommendations on when different indices should be reported are provided. Translational Abstract Reporting of reliability information of measures on the studied sample is a basic requirement in psychological research. For multilevel data-data that have a hierarchical structure such as students nested within schools and survey participants nested within neighborhoods, previous research commonly used the between-level and within-level composite reliability proposed by Geldhof, Preacher, and Zyphur (2014). However, this article shows that the previously proposed between-level composite reliability can provide overly optimistic reliability coefficient because it ignores one major source of error, namely the sampling error of cluster means. To obtain more accurate reliability information for multilevel data, this article proposes alternative reliability indices that correctly account for the different sources of measurement error. The computation of the between-level indices also takes into account whether the construct being measured is an aggregate of individual characteristics (e.g., mean student achievement of a school) or an inherent group-level characteristic (e.g., school climate). I illustrate the proposed indices using a large-scale national data set with four items measuring students' attitudes toward mathematics. Software code in the R and the Mplus software was provided so that applied researchers can use to compute the proposed indices for their data and the corresponding confidence intervals.