Analyzing decision boundaries of neural networks

In this peeper, we analyze decision boundaries of 3 layer feedforward neural networks that use the sigmoid function as an activation function. By analyzing the decision boundaries in the space defined by the outputs of the hidden neurons, we found that the decision boundaries are always linear boundaries and that the decision boundaries are not completely independent. We found that for a 3-pattern class problem, the decision boundaries in the space defined by the outputs of the hidden neurons should meet at the same intersection. And this dependency of decision boundaries is extended to multiclass problems, providing valuable insight into decision boundaries. In particular for a K-pattern classes problems, we found that there are only K-l degree of freedoms in drawing decision boundaries in the space defined by the outputs of the hidden neurons, though there are C-K(2) decision boundaries. Finally, we present some interesting examples of decision boundaries of neural networks.