Network generalization differences quantified

It has long been observed, and frequently noted, by connectionists that small changes in initial conditions, prior to training, can result in networks that generalize very differently. We have performed a systematic study of this phenomenon, using a number of different statistical measures of generalization differences. From these we derive a formal definition of Generalization Diversity. We quantify the relative impacts on generalization of the major parameters used in network initialization as well as extend the formal framework to also encompass the differences in generalization difference from one parameter to another. We reveal, for example, the relative effects of random initialization of the link weights and variation of the number of hidden units, and how similar these two resultant effects are. Finally, examples are presented of how the proposed generalization diversity measure may be exploited in order to improve the performance of neural-net systems. We show how several of these measures can be used to engineer reliability improvements in neural-net systems.