APPLES TO ORANGES OR GALA VERSUS GOLDEN DELICIOUS? COMPARING DATA QUALITY OF NONPROBABILITY INTERNET SAMPLES TO LOW RESPONSE RATE PROBABILITY SAMPLES

被引:66
作者
Dutwin, David [1 ]
Buskirk, Trent D. [2 ,3 ,4 ]
机构
[1] SSRS Social Sci Res Solut, Media, PA USA
[2] Univ Massachusetts, Management Sci & Informat Syst Data Sci, Boston, MA 02125 USA
[3] Univ Massachusetts, Survey Res Ctr, Boston, MA 02125 USA
[4] Mkt Syst Grp MSG, Stat & Methodol, Horsham, PA USA
关键词
NONRESPONSE RATES; ONLINE; DESIGN; BIAS;
D O I
10.1093/poq/nfw061
中图分类号
G2 [信息与知识传播];
学科分类号
05 ; 0503 ;
摘要
Nonprobability samples have gained mass popularity and use in many research circles, including market research and some political research. One justification for the use of nonprobability samples is that low response rate probability surveys have nothing significant to offer over and above a "well built" nonprobability sample. Utilizing an elemental approach, we compare a range of samples, weighting, and modeling procedures in an analysis that evaluates the estimated bias of various cross-tabulations of core demographics. Specifically, we compare a battery of bias related metrics for nonprobability panels, dual-frame telephone samples, and a high-quality in-person sample. Results indicate that there is roughly a linear trend, with nonprobability samples attaining the greatest estimated bias, and the in-person sample, the lowest. Results also indicate that the bias estimates vary widely for the nonprobability samples compared to either the telephone or in-person samples, which themselves tend to have consistently smaller amounts of estimated bias. Specifically, both weighted and unweighted dual-frame telephone samples were found to have about half the estimated bias compared to analogous nonprobability samples. Advanced techniques such as propensity weighting and sample matching did not improve these measures, and in some cases made matters worse. Implications for "fit for purpose" in survey research are discussed given these findings.
引用
收藏
页码:213 / 239
页数:27
相关论文
共 38 条
[1]  
[Anonymous], 2012, [No title captured]
[3]   Total Survey Error: Design, Implementation, and Evaluation [J].
Biemer, Paul P. .
PUBLIC OPINION QUARTERLY, 2010, 74 (05) :817-848
[4]  
Brick J., 2015, AAPOR C HOLL FL US
[5]  
Buskirk T., 2012, JSM P, P3696
[6]  
Buskirk Trent, 2015, AAPOR C HOLL FL US
[7]  
Buskirk Trent, 2015, INT TOT SURV ERR C B
[8]  
Callegaro M, 2014, WILEY SER SURV METH, P23
[9]   National Surveys Via Rdd Telephone Interviewing Versus the Internet [J].
Chang, Linchiat ;
Krosnick, Jon A. .
PUBLIC OPINION QUARTERLY, 2009, 73 (04) :641-678
[10]   Comparison of US Panel Vendors for Online Surveys [J].
Craig, Benjamin M. ;
Hays, Ron D. ;
Pickard, A. Simon ;
Cella, David ;
Revicki, Dennis A. ;
Reeve, Bryce B. .
JOURNAL OF MEDICAL INTERNET RESEARCH, 2013, 15 (11)