Regularization and averaging of the selective Naive Bayes classifier

The Naive Bayes classifier has proved to be very effective on many real data applications. Its performances usually benefit from an accurate estimation of univariate conditional probabilities and from variable selection. However, although variable selection is a desirable feature, it is prone to overfitting. In this paper, we introduce a new regularization technique to select the most probable subset of variables and propose a new model averaging method. The weighting scheme on the models reduces to a weighting scheme on the variables, and finally results in a Naive Bayes with "soft variable selection". Extensive experimental results show that the averaged regularized classifier outperforms the initial Selective Naive Bayes classifier.