On the relative importance of covariate choice, reliable measurement and analytical method for estimating causal effects from observational studies

In evaluation studies it is often not possible to conduct randomized experiments for estimating causal effects of a program. Instead, quasi-experimental designs are frequently used. The causal interpretation of a statistically adjusted estimate is only warranted if the assumptions of the underlying quasi-experiment are met and the statistical analysis is properly conducted. In this article the relative importance of covariate choice, the covariates' reliable measurement, and the choice of analytic method (regression analysis or diffent types of propensity score methods) are investigated using a within-study comparison which compares the effect estimate from adjusted quasi-experimental design to the corresponding effect estimate from a randomized experiment. The results show that the covariate choice is the most important factor for removing selection bias. The reliable measurement of covariates is the second most important factor. The choice of an analytic method does not have a strong impact on bias reduction. To date, the superiority of propensity score methods in comparison to regression methods is not yet supported by empirical evidence.