Estimating and Testing for Differential Treatment Effects on Outcomes When the Outcome Variances Differ

Researchers working in the context of randomized trials routinely estimate and test for treatment effects on the study outcomes. This article discusses the merits of assessing differential treatment effects across outcomes and proposes a multivariate approach using standardized outcomes for this purpose. This multivariate approach extends prior approaches to an arbitrary number of treatment groups and outcomes and does not require that the within-group covariance matrix have particular properties (e.g., sphericity). Theoretical analyses articulate the inferential basis for earlier recommendations for data standardization prior to analysis and demonstrate that inferential procedures (e.g., null hypothesis significance tests and confidence intervals) can exhibit poor operating characteristics when unstandardized outcome data are used for analysis and differential standardized treatment effects are the conceptual, intended target of inference. This article explores these and other issues (e.g., statistical power to detect and confidence intervals for differential standardized treatment effects) and demonstrates the proposed approach using data from a published experiment. The theoretical utility of differential treatment-effect evidence is considered from a construct validity perspective for randomized trials. The proposed approach provides inferential procedures to evaluate theoretically motivated predictions for differential treatment effects on the outcomes; failure to support such predictions either calls the construct validity of the randomized trial into question or the underlying theory. The proposed approach also enables the detection of differences in treatment effects on the outcomes that are not theoretically expected; such results, especially if replicated, would motivate the need for theoretical refinements.