Stateful MapReduce Framework for mRMR Feature Selection Using Horizontal Partitioning

Feature selection (FS) is an important pre-processing step in building machine learning models. minimum Redundancy and Maximum Relevance (mRMR) approach has emerged as one of the successful algorithms in obtaining irredundant feature subset involving only bivariate computations. In the current digital age, owing to the prevalence of very large scale datasets, an imminent need has arisen for scalable solutions using distributed/parallel algorithms. MapReduce solutions are proven to be one of the best approaches to design fault-tolerant and scalable solutions. This work analyses the existing Horizontal MapReduce approaches for mRMR feature selection and identifies the limitations thereof. It is observed that existing approaches involve redundant and repetitive computations and lacks a metadata framework to diminish them. This motivated us to propose Horizontal partitioning based MapReduce solutions namely HMR_mRMR, is an Iterative MapReduce algorithms and is designed under Apache Spark. Appropriate usage of metadata framework and solution formulation optimizes the computations in the proposed approaches. The comparative experimental study is conducted with existing approaches to establish the importance of HMR_mRMR.