On minimal representations of shallow ReLU networks

The realization function of a shallow ReLU network is a continuous and piecewise affine function f : R-d ->& nbsp;R, where the domain Rd is partitioned by a set of n hyperplanes into cells on which f is affine. We show that the minimal representation for f uses either n, n + 1 or n + 2 neurons and we characterize each of the three cases. In the particular case, where the input layer is one-dimensional, minimal representations always use at most n+1 neurons but in all higher dimensional settings there are functions for which n+2 neurons are needed. Then we show that the set of minimal networks representing f forms a C-infinity-submanifold M and we derive the dimension and the number of connected components of M. Additionally, we give a criterion for the hyperplanes that guarantees that a continuous, piecewise affine function is the realization function of an appropriate shallow ReLU network.(c) 2022 Elsevier Ltd. All rights reserved.