USING RANDOM WEIGHTS TO TRAIN MULTILAYER NETWORKS OF HARD-LIMITING UNITS

A gradient descent algorithm suitable for training multilayer feedforward networks of processing units with hard-limiting output functions is presented. The conventional back-propagation algorithm cannot be applied to networks whose processing units have hard-limiting input-output characteristics because the required derivatives are not available. However, if the network weights are random variables with smooth distribution functions, the probability of a hard-limiting unit taking one of its two possible values is a continuously differentiable function. In this paper, we use this to develop an algorithm similar to back-propagation, but for the hard-limiting case. It is shown that the computational framework of this algorithm is similar to standard back-propagation, but there is an additional computational expense involved in the estimation of gradients. We give upper bounds on this estimation penalty. Two examples are given which indicate that, when this algorithm is used to train networks of hard-limiting units, its performance is similar to that of conventional back-propagation applied to networks of units with sigmoidal characteristics.