Variable selection via a combination of the L0 and L1 penalties

Variable selection is an important aspect of high-dimensional statistical modeling, particularly in regression and classification. In the regularization framework, various penalty functions are used to perform variable selection by putting relatively large penalties on small coefficients. The L-1 penalty is a popular choice because of its convexity, but it produces biased estimates for the large coefficients. The L-0 penalty is attractive for variable selection because it directly penalizes the number of nonzero coefficients. However, the optimization involved is discontinuous and nonconvex, and therefore it is very challenging to implement. Moreover, its solution may not be stable. In this article, we propose a new penalty that combines the L-0 and L-1 penalties. We implement this new penalty by developing a global optimization algorithm using mixed integer programming (MIP). We compare this combined penalty with several other penalties via simulated examples as well as real applications. The results show that the new penalty outperforms both the L-0 and L-1 penalties in terms of variable selection while maintaining good prediction accuracy.