The Comparative Analysis of Single-Objective and Multi-objective Evolutionary Feature Selection Methods

This research presents a comparative analysis of single-objective and multi-objective evolutionary feature selection methods over interpretable models. The question taken in this research is to investigate the role of aforementioned techniques for feature selection on classification model's interpretability as well as accuracy. Since, feature selection is a non-deterministic polynomial-time hardness (NP-hard) problem therefore exhaustively searching for all the possible feature sets is not computationally feasible. Evolutionary algorithms provide a very powerful searching mechanism that is utilized for candidate feature generation in a reasonable time frame. Single-objective (SO) algorithms are generally geared towards finding a subset of candidate feature set which achieves highest evaluation score e.g. classification accuracy. On the other hand, multi-objective (MO) methods are relatively more comprehensive than their counterparts. MO feature selection algorithms can simultaneously optimize two or more objectives such as classification accuracy of a final feature set along with the cardinality of the feature set. In this research, we have selected two representative feature selection algorithms from both the groups. Decision tree and a rule-based classifiers are used for the performance evaluation in terms of interpretability i.e. model size, and predictive accuracy. This research is undertaken to investigate application of SO and MO feature selection methods on small to medium sized classification datasets. The experimental results on 3 interpretable classification models indicate that the relative differences between the two set of models on small datasets may not be much pronounced, yet for the medium-sized datasets MO models provide a promising alternative. Although multi-objective techniques resulted in smaller feature subsets in general, but overall the difference between both the single-objective and the multi-objective feature subset selection techniques, investigated in this study, is not statistical significant.