Improved Confidence Intervals for Differences Between Standardized Effect Sizes

An evaluation of a difference between effect sizes from two dependent variables in a single study is likely to be based on differences between standard scores if raw scores on those variables are not scaled in comparable units of measurement. The standardization used for this purpose is usually sample-based rather than population-based, but the consequences of this distinction for the construction of confidence intervals on differential effects have not been systematically examined. In this article I show that differential effect confidence intervals (CIs) constructed from differences between the standard scores produced by sample-based standardization can be too narrow when those effects are large and dependent variables are highly correlated, particularly in within-subjects designs. I propose a new approach to the construction of differential effect CIs based on differences between adjusted sample-based standard scores that allow conventional CI procedures to produce Bonett-type CIs (Bonett, 2008) on individual effects. Computer simulations show that differential effect CIs constructed from adjusted standard scores can provide much better coverage probabilities than CIs constructed from unadjusted standard scores.