Order Selection in Finite Mixture Models With a Nonsmooth Penalty

Order selection is a fundamental and challenging problem in the application of finite mixture models. In this article, we develop a new penalized likelihood approach. The new method, modified smoothly clipped absolute deviation (MSCAD), deviates from information-based methods such as Akaike information criterion (AIC) and Bayesian information criterion (BIC) by introducing two penalty functions that depend on the mixing proportions and the component parameters. It is consistent at estimating both the order of the mixture model and the mixing distribution. Simulations show that MSCAD has much better performance than a number of existing methods. Two real-data examples are examined to illustrate the performance of MSCAD.