Orthogonal Super Greedy Learning for Sparse Feedforward Neural Networks

The analytic approaches for feedforward neural network, e.g., Radial Basis Function (RBF), have some attractive characteristics, such as superior theoretical properties and faster numerical implementations. However, they still have several drawbacks. The primary defect is that their generation performance and computational complexity are susceptible to the influence of irrelevant hidden variables. Thus, how to alleviate this influence has become a crucial issue for the feedforward neural network. In this paper, we propose an Orthogonal Super Greedy learning (OSGL) method for hidden neurons selection. The OSGL selects more than one hidden neurons from a given network structure in a greedy strategy until an adequate sparse network has been constructed. Theoretical analyses show it can reach the optimal learning rate. Extensive empirical results demonstrate the superiority that the proposed method can produce an excellent generalization performance with a sparse and compact feature representation within feedforward networks.