Increasing and Decreasing Returns and Losses in Mutual Information Feature Subset Selection

被引:6
作者
Van Dijck, Gert [1 ]
Van Hulle, Marc M. [1 ]
机构
[1] Katholieke Univ Leuven, Lab Neuro Psychofysiol, Computat Neurosci Res Grp, B-3000 Louvain, Belgium
关键词
Bayesian networks; bit parity; conditional entropy; conditional mutual information; decreasing losses; decreasing returns; feature subset selection; increasing losses; increasing returns; INPUT FEATURE-SELECTION; PROBABILITY; ERROR; CLASSIFICATION; ALGORITHM; RELEVANCE; ENTROPY;
D O I
10.3390/e12102144
中图分类号
O4 [物理学];
学科分类号
0702 ;
摘要
Mutual information between a target variable and a feature subset is extensively used as a feature subset selection criterion. This work contributes to a more thorough understanding of the evolution of the mutual information as a function of the number of features selected. We describe decreasing returns and increasing returns behavior in sequential forward search and increasing losses and decreasing losses behavior in sequential backward search. We derive conditions under which the decreasing returns and the increasing losses behavior hold and prove the occurrence of this behavior in some Bayesian networks. The decreasing returns behavior implies that the mutual information is concave as a function of the number of features selected, whereas the increasing returns behavior implies this function is convex. The increasing returns and decreasing losses behavior are proven to occur in an XOR hypercube.
引用
收藏
页码:2144 / 2170
页数:27
相关论文
共 35 条
[1]  
[Anonymous], 1991, ELEMENTS INFORM THEO, DOI [DOI 10.1002/0471200611, 10.1002/0471200611]
[2]  
[Anonymous], 2001, Pattern Classification
[3]  
[Anonymous], 2010, P 27 INT C MACHINE L
[4]   USING MUTUAL INFORMATION FOR SELECTING FEATURES IN SUPERVISED NEURAL-NET LEARNING [J].
BATTITI, R .
IEEE TRANSACTIONS ON NEURAL NETWORKS, 1994, 5 (04) :537-550
[5]   Feature selection, mutual information, and the classification of high-dimensional patterns [J].
Bonev, Boyan ;
Escolano, Francisco ;
Cazorla, Miguel .
PATTERN ANALYSIS AND APPLICATIONS, 2008, 11 (3-4) :309-319
[6]   Normalized Mutual Information Feature Selection [J].
Estevez, Pablo. A. ;
Tesmer, Michel ;
Perez, Claudio A. ;
Zurada, Jacek A. .
IEEE TRANSACTIONS ON NEURAL NETWORKS, 2009, 20 (02) :189-201
[7]  
FANO R, 1996, TRANSMISSION INFORM
[8]   RELATIONS BETWEEN ENTROPY AND ERROR-PROBABILITY [J].
FEDER, M ;
MERHAV, N .
IEEE TRANSACTIONS ON INFORMATION THEORY, 1994, 40 (01) :259-266
[9]  
Fleuret F, 2004, J MACH LEARN RES, V5, P1531
[10]   Resampling methods for parameter-free and robust feature selection with mutual information [J].
Francois, D. ;
Rossi, F. ;
Wertz, V. ;
Verleysen, M. .
NEUROCOMPUTING, 2007, 70 (7-9) :1276-1288