A new ranking-based stability measure for feature selection algorithms

The stability of a feature selection (FS) algorithm is one of the most crucial issues when working with a machine learning model. Until now, various stability measures based on a subset of features have been proposed. However, they lack consideration for feature ranking which is equally important to judge the robustness of algorithms. This paper proposes a novel frequency-based stability measure called rank stability (RSt) that evaluates FS algorithms on both criteria, i.e., subsets of features and feature rankings. The proposed measure evaluates the variation of feature rankings generated by FS algorithms after making a small perturbation to the training set. We mathematically justify the proposed measure based on the earlier and newly defined desirable properties. Additionally, we explore various heterogeneous ensemble techniques and compare them with traditional FS algorithms on real-world datasets. We perform extensive experiments to demonstrate that the heterogeneous ensemble techniques perform better than traditional FS algorithms with respect to the proposed measure and other performance metrics.