The Impact of Regularization on High-dimensional Logistic Regression

被引:0
|
作者
Salehi, Fariborz [1 ]
Abbasi, Ehsan [1 ]
Hassibi, Babak [1 ]
机构
[1] CALTECH, Dept Elect Engn, Pasadena, CA 91125 USA
来源
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019) | 2019年 / 32卷
基金
美国国家科学基金会;
关键词
GENERALIZED LINEAR-MODELS; SELECTION;
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Logistic regression is commonly used for modeling dichotomous outcomes. In the classical setting, where the number of observations is much larger than the number of parameters, properties of the maximum likelihood estimator in logistic regression are well understood. Recently, Sur and Candes [26] have studied logistic regression in the high-dimensional regime, where the number of observations and parameters are comparable, and show, among other things, that the maximum likelihood estimator is biased. In the high-dimensional regime the underlying parameter vector is often structured (sparse, block-sparse, finite-alphabet, etc.) and so in this paper we study regularized logistic regression (RLR), where a convex regularizer that encourages the desired structure is added to the negative of the log-likelihood function. An advantage of RLR is that it allows parameter recovery even for instances where the (unconstrained) maximum likelihood estimate does not exist. We provide a precise analysis of the performance of RLR via the solution of a system of six nonlinear equations, through which any performance metric of interest (mean, mean-squared error, probability of support recovery, etc.) can be explicitly computed. Our results generalize those of Sur and Candes and we provide a detailed study for the cases of l(2)(2)-RLR and sparse (l(1)-regularized) logistic regression. In both cases, we obtain explicit expressions for various performance metrics and can find the values of the regularizer parameter that optimizes the desired performance. The theory is validated by extensive numerical simulations across a range of parameter values and problem instances.
引用
收藏
页数:11
相关论文
共 50 条
  • [1] CONVEX REGULARIZATION FOR HIGH-DIMENSIONAL MULTIRESPONSE TENSOR REGRESSION
    Raskutti, Garvesh
    Yuan, Ming
    Chen, Han
    ANNALS OF STATISTICS, 2019, 47 (03) : 1554 - 1584
  • [2] Model Selection for High-Dimensional Quadratic Regression via Regularization
    Hao, Ning
    Feng, Yang
    Zhang, Hao Helen
    JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION, 2018, 113 (522) : 615 - 625
  • [3] On inference in high-dimensional logistic regression models with separated data
    Lewis, R. M.
    Battey, H. S.
    BIOMETRIKA, 2024, 111 (03) : 989 - 1011
  • [4] High-Dimensional Multivariate Linear Regression with Weighted Nuclear Norm Regularization
    Suh, Namjoon
    Lin, Li-Hsiang
    Huo, Xiaoming
    JOURNAL OF COMPUTATIONAL AND GRAPHICAL STATISTICS, 2024, 33 (04) : 1264 - 1275
  • [5] Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models
    Ma, Rong
    Cai, T. Tony
    Li, Hongzhe
    JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION, 2021, 116 (534) : 984 - 998
  • [6] STATISTICAL INFERENCE FOR GENETIC RELATEDNESS BASED ON HIGH-DIMENSIONAL LOGISTIC REGRESSION
    Ma, Rong
    Guo, Zijian
    Cai, T. Tony
    Li, Hongzhe
    STATISTICA SINICA, 2024, 34 (02) : 1023 - 1043
  • [7] Robust and sparse estimation methods for high-dimensional linear and logistic regression
    Kurnaz, Fatma Sevinc
    Hoffmann, Irene
    Filzmoser, Peter
    CHEMOMETRICS AND INTELLIGENT LABORATORY SYSTEMS, 2018, 172 : 211 - 222
  • [8] Debiased inference for heterogeneous subpopulations in a high-dimensional logistic regression model
    Kim, Hyunjin
    Lee, Eun Ryung
    Park, Seyoung
    SCIENTIFIC REPORTS, 2023, 13 (01)
  • [9] Interpretation of high-dimensional linear regression: Effects of nullspace and regularization demonstrated on battery data
    Schaeffer, Joachim
    Lenz, Eric
    Chueh, William C.
    Bazant, Martin Z.
    Findeisen, Rolf
    Braatz, Richard D.
    COMPUTERS & CHEMICAL ENGINEERING, 2024, 180
  • [10] Using synthetic data and dimensionality reduction in high-dimensional classification via logistic regression
    Zarei, Shaho
    Mohammadpour, Adel
    COMPUTATIONAL METHODS FOR DIFFERENTIAL EQUATIONS, 2019, 7 (04): : 626 - 634