Comparison of Reliability Measures Under Factor Analysis and Item Response Theory

Reliability of test scores is one of the most pervasive psychometric concepts in measurement. Reliability coefficients based on a unifactor model for continuous indicators include maximal reliability rho and an unweighted sum score-based omega, among many others. With increasing popularity of item response theory, a parallel reliability measure pi has been introduced using the information function. This article studies the relationship among the three reliability coefficients. Exploiting the equivalency between item factor analysis and the normal ogive model, pi((2)) for dichotomous data is shown to be always smaller than rho. Additional results imply that omega is typically greater than pi((2)) in practical conditions, though mathematically there is no dominant relationship between pi((2)) and omega. Furtherresults indicate that, as the number of response categories increases, pi can surpass omega. The reasons why pi and omega fall short of rho are also explored from an information gain/loss perspective. Implications of the findings on scale development and analysis are discussed.