Risk bounds for model selection via penalization

被引:382
作者
Barron, A
Birgé, L
Massart, P
机构
[1] Yale Univ, Dept Stat, New Haven, CT 06520 USA
[2] Univ Paris 06, Probabil Lab, CNRS, URA Stat & Modeles Aleatoires 1321, F-75252 Paris 05, France
[3] Univ Paris 11, CNRS, URA Modelisat Stochast & Stat 743, F-91405 Orsay, France
关键词
penalization; model selection; adaptive estimation; empirical processes; sieves; minimum contrast estimators;
D O I
10.1007/s004400050210
中图分类号
O21 [概率论与数理统计]; C8 [统计学];
学科分类号
020208 ; 070103 ; 0714 ;
摘要
Performance bounds for criteria for model selection are developed using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term motivated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of observations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penalized empirical contrast estimator is bounded by an index of the accuracy of the sieve. This accuracy index quantifies the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size. If we choose a list of models which exhibit good approximation properties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approximately adaptive (typically up to a slowly varying function of the sample size). We shall provide various illustrations of our method such as penalized maximum likelihood projection or least squares estimation. The models will involve commonly used finite dimensional expansions such as piecewise polynomials with fixed or variable knots, trigonometric polynomials, wavelets, neural nets and related nonlinear expansions defined by superposition of ridge functions.
引用
收藏
页码:301 / 413
页数:113
相关论文
共 50 条
  • [21] Simultaneous Model Selection and Feature Selection via BYY Harmony Learning
    Wang, Hongyan
    Ma, Jinwen
    [J]. ADVANCES IN NEURAL NETWORKS - ISNN 2011, PT II, 2011, 6676 : 47 - +
  • [22] PENALIZATION IN NONCLASSICAL CONVEX-PROGRAMMING VIA VARIATIONAL CONVERGENCE
    ALART, P
    LEMAIRE, B
    [J]. MATHEMATICAL PROGRAMMING, 1991, 51 (03) : 307 - 331
  • [23] Variable Selection in Nonparametric Classification Via Measurement Error Model Selection Likelihoods
    Stefanski, L. A.
    Wu, Yichao
    White, Kyle
    [J]. JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION, 2014, 109 (506) : 574 - 589
  • [24] Developing biomarker combinations in multicenter studies via direct maximization and penalization
    Meisner, Allison
    Parikh, Chirag R.
    Kerr, Kathleen F.
    [J]. STATISTICS IN MEDICINE, 2020, 39 (24) : 3412 - 3426
  • [25] MODEL SELECTION VIA MULTIFOLD CROSS-VALIDATION
    ZHANG, P
    [J]. ANNALS OF STATISTICS, 1993, 21 (01) : 299 - 313
  • [26] Variogram model selection via nonparametric derivative estimation
    Gorsich, DJ
    Genton, MG
    [J]. MATHEMATICAL GEOLOGY, 2000, 32 (03): : 249 - 270
  • [27] Model selection via Genetic Algorithms for RBF networks
    de Lacerda, EGM
    de Carvalho, ACPLF
    Ludermir, TB
    [J]. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS, 2002, 13 (2-4) : 111 - 122
  • [28] Variogram Model Selection via Nonparametric Derivative Estimation
    David J. Gorsich
    Marc G. Genton
    [J]. Mathematical Geology, 2000, 32 : 249 - 270
  • [29] Compact Face Representation via Forward Model Selection
    Shao, Weiyuan
    Wang, Hong
    Zheng, Yingbin
    Ye, Hao
    [J]. BIOMETRIC RECOGNITION, 2016, 9967 : 112 - 120
  • [30] Causes of effects via a Bayesian model selection procedure
    Corradi, Fabio
    Musio, Monica
    [J]. JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES A-STATISTICS IN SOCIETY, 2020, 183 (04) : 1777 - 1792