Why We Should Not Be Indifferent to Specification Choices for Difference-in-Differences

被引:242
作者
Ryan, Andrew M. [1 ]
Burgess, James F., Jr. [2 ]
Dimick, Justin B. [3 ]
机构
[1] Univ Michigan, Sch Publ Hlth, Ann Arbor, MI 48109 USA
[2] Boston Univ, Sch Publ Hlth, US Dept Vet Affairs, Vet Affairs Boston Hlth Care Syst, Boston, MA USA
[3] Univ Michigan, Sch Med, Dept Surg, Ctr Healthcare Outcomes & Policy, Ann Arbor, MI USA
基金
美国医疗保健研究与质量局;
关键词
Hospitals; econometrics; health economics; quality of care; health policy; HEALTH-INSURANCE COVERAGE; PAY-FOR-PERFORMANCE; BARIATRIC SURGERY; MEDICARE BENEFICIARIES; FINANCIAL INCENTIVES; QUALITY IMPROVEMENT; MENTAL-HEALTH; HOSPITAL PAY; REPORT CARDS; PANEL-DATA;
D O I
10.1111/1475-6773.12270
中图分类号
R19 [保健组织与事业(卫生事业管理)];
学科分类号
摘要
ObjectiveTo evaluate the effects of specification choices on the accuracy of estimates in difference-in-differences (DID) models. Data SourcesProcess-of-care quality data from Hospital Compare between 2003 and 2009. Study DesignWe performed a Monte Carlo simulation experiment to estimate the effect of an imaginary policy on quality. The experiment was performed for three different scenarios in which the probability of treatment was (1) unrelated to pre-intervention performance; (2) positively correlated with pre-intervention levels of performance; and (3) positively correlated with pre-intervention trends in performance. We estimated alternative DID models that varied with respect to the choice of data intervals, the comparison group, and the method of obtaining inference. We assessed estimator bias as the mean absolute deviation between estimated program effects and their true value. We evaluated the accuracy of inferences through statistical power and rates of false rejection of the null hypothesis. Principal FindingsPerformance of alternative specifications varied dramatically when the probability of treatment was correlated with pre-intervention levels or trends. In these cases, propensity score matching resulted in much more accurate point estimates. The use of permutation tests resulted in lower false rejection rates for the highly biased estimators, but the use of clustered standard errors resulted in slightly lower false rejection rates for the matching estimators. ConclusionsWhen treatment and comparison groups differed on pre-intervention levels or trends, our results supported specifications for DID models that include matching for more accurate point estimates and models using clustered standard errors or permutation tests for better inference. Based on our findings, we propose a checklist for DID analysis.
引用
收藏
页码:1211 / 1235
页数:25
相关论文
共 66 条
[1]   Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California's Tobacco Control Program [J].
Abadie, Alberto ;
Diamond, Alexis ;
Hainmueller, Jens .
JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION, 2010, 105 (490) :493-505
[2]   Interaction terms in logit and probit models [J].
Ai, CR ;
Norton, EC .
ECONOMICS LETTERS, 2003, 80 (01) :123-129
[3]   One last puff? Public smoking bans and smoking behavior [J].
Anger, Silke ;
Kvasnicka, Michael ;
Siedler, Thomas .
JOURNAL OF HEALTH ECONOMICS, 2011, 30 (03) :591-601
[4]  
Angrist JD, 2009, MOSTLY HARMLESS ECONOMETRICS: AN EMPIRICISTS COMPANION, P1
[5]   The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics [J].
Angrist, Joshua D. ;
Pischke, Joern-Steffen .
JOURNAL OF ECONOMIC PERSPECTIVES, 2010, 24 (02) :3-30
[6]   Identification and inference in nonlinear difference-in-differences models [J].
Athey, S ;
Imbens, GW .
ECONOMETRICA, 2006, 74 (02) :431-497
[7]   Impact of full mental health and substance abuse parity for children in the Federal Employees Health Benefits Program [J].
Azrin, Susan T. ;
Huskamp, Haiden A. ;
Azzone, Vanessa ;
Goldman, Howard H. ;
Frank, Richard G. ;
Burnam, M. Audrey ;
Normand, Sharon-Lise T. ;
Ridgely, M. Susan ;
Young, Alexander S. ;
Barry, Colleen L. ;
Busch, Alisa B. ;
Moran, Garrett .
PEDIATRICS, 2007, 119 (02) :E452-E459
[8]   Effect of influenza vaccination on hospitalizations in persons aged 50 years and older [J].
Baxter, Roger ;
Ray, G. Thomas ;
Fireman, Bruce H. .
VACCINE, 2010, 28 (45) :7267-7272
[9]   How much should we trust differences-in-differences estimates? [J].
Bertrand, M ;
Duflo, E ;
Mullainathan, S .
QUARTERLY JOURNAL OF ECONOMICS, 2004, 119 (01) :249-275
[10]   Can labor regulation hinder economic performance? Evidence from India [J].
Besley, T ;
Burgess, R .
QUARTERLY JOURNAL OF ECONOMICS, 2004, 119 (01) :91-134