Non-linear feature extraction by redundancy reduction in an unsupervised stochastic neural network

Unsupewised feature extraction by a stochastic neural network can be defined as a minimization of the redundancy between the elements ofthe output layer, given complete information transferfrom input to output. Redundancy minimization can be achieved by minimization of the mutual infonnation between the units of the output layer. Complete information transfer is enforced by maximizing the mutual information ofthe input and output. With these two conditions we define a novel learning algorithmfor stochastic recurrent networks. The minimum ofredundancy corresponds to the extraction ofstatistically independentfeatures, leading to afactorial representation ofthe environment. The resulting learning rule includes Hebbian and anti-Hebbian learning tenns, These two terms are weighted by the amount of information transmitted in the learning synapse minus the grade of redundant information in the corresponding output neuron, giving thus, an information-theoretic interpretation of the proportionality constant of Hebb 's biological rule. Simulations demonstrate the performance of this method. When a retina is simulated, the learning algorithm forms decorrelated receptive fields, This represents the first experiment that extends the results of the linear principle component analysis to the nonlinear case by a direct implementation of Barlow 's principle of redundancy reductionfor unsupewisedfeatures extraction by receptive fields formation in a retina model. (C) 1997 Elsevier Science Ltd.