Moving beyond Alpha: A Primer on Alternative Sources of Single-Administration Reliability Evidence for Quantitative Chemistry Education Research

This methodological paper examines current conceptions of reliability in chemistry education research (CER) and provides recommendations for moving beyond the current reliance on reporting coefficient alpha (alpha) as reliability evidence without regard to its appropriateness for the research context. To help foster a better understanding of reliability and the assumptions that underlie reliability coefficients, reliability is first described from a conceptual framework, drawing on examples from measurement in the physical sciences; then classical test theory is used to frame a discussion of how reliability evidence for psychometric measurements is commonly examined in CER, primarily in the form of single-administration reliability coefficients. Following this more conceptual introduction to reliability, the paper transitions to a more mathematical treatment of reliability using a factor analysis framework with emphasis on the assumptions underlying coefficient alpha and other single-administration reliability coefficients, such as omega (omega) and coefficient H, which are recommended as successors to alpha in CER due to their more broad applicability to a variety of factor models. The factor analysis-based reliability discussion is accompanied by R code that demonstrates the mathematical relations underlying single-administration reliability coefficients and provides interested readers the opportunity to compute coefficients beyond alpha for their own data.