Difficulty of selecting among multilevel models using predictive accuracy

被引:8
作者
Wang, Wei [1 ]
Gelman, Andrew [2 ]
机构
[1] Columbia Univ, Dept Stat, New York, NY 10027 USA
[2] Columbia Univ, Dept Stat & Polit Sci, New York, NY 10027 USA
关键词
Multilevel models; Predictive accuracy; Model selection; Sample survey; Cross-validation; CROSS-VALIDATION;
D O I
10.4310/SII.2015.v8.n2.a3
中图分类号
Q [生物科学];
学科分类号
07 ; 0710 ; 09 ;
摘要
As a simple and compelling approach for estimating outof-sample prediction error, cross-validation naturally lends itself to the task of model comparison. However, even with moderate sample size, it can be surprisingly difficult to compare multilevel models based on predictive accuracy. Using a hierarchical model fit to large survey data with a battery of questions, we demonstrate that even though cross-validation might give good estimates of pointwise out-of-sample prediction error, it is not always a sensitive instrument for model comparison.
引用
收藏
页码:153 / 160
页数:8
相关论文
共 24 条
  • [1] [Anonymous], 2011, INNOVATIONS COMPUTER
  • [2] Fitting Linear Mixed-Effects Models Using lme4
    Bates, Douglas
    Maechler, Martin
    Bolker, Benjamin M.
    Walker, Steven C.
    [J]. JOURNAL OF STATISTICAL SOFTWARE, 2015, 67 (01): : 1 - 48
  • [3] A CROSS-VALIDATORY METHOD FOR DEPENDENT DATA
    BURMAN, P
    CHOW, E
    NOLAN, D
    [J]. BIOMETRIKA, 1994, 81 (02) : 351 - 358
  • [4] BURMAN P, 1989, BIOMETRIKA, V76, P503, DOI 10.2307/2336116
  • [5] How Does Multilevel Regression and Poststratification Perform with Conventional National Surveys?
    Buttice, Matthew K.
    Highton, Benjamin
    [J]. POLITICAL ANALYSIS, 2013, 21 (04) : 449 - 467
  • [6] A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models
    Chung, Yeojin
    Rabe-Hesketh, Sophia
    Dorie, Vincent
    Gelman, Andrew
    Liu, Jingchen
    [J]. PSYCHOMETRIKA, 2013, 78 (04) : 685 - 709
  • [7] Craven P., 1979, Numerische Mathematik, V31, P377, DOI 10.1007/BF01404567
  • [8] Dorie Vincent., 2013, blme: Bayesian Linear Mixed-Effects Models
  • [9] ESTIMATES OF INCOME FOR SMALL PLACES - APPLICATION OF JAMES-STEIN PROCEDURES TO CENSUS-DATA
    FAY, RE
    HERRIOT, RA
    [J]. JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION, 1979, 74 (366) : 269 - 277
  • [10] Gelman A, 2010, RED STATE, BLUE STATE, RICH STATE, POOR STATE: WHY AMERICANS VOTE THE WAY THEY DO, EXPANDED EDITION, P1