On model complexity and selection

In the past, practitioners and researchers have compared the performance of neural networks with other model classes based on the multiple correlation coefficient or empirical validation. Such comparisons are biased towards neural networks as. such performance metrics do not account for model complexity. Model complexity metrics are essential for parameter significance (e.g., F-test) and model mis-specification tests (e.g., autocorrelation). The estimation of degrees of freedom from the projection matrix of regression is therefore vitally important in all phases of the model building process for neural regression models. Degrees of freedom are used to measure model complexity and thus adjust statistics so that they may be meaningfully applied to regression models. In this paper we derive expressions for the influence matrix in linear and non-linear regression models and non-linear models with regularization, including neural networks. We show that they can be obtained within the same framework. In particular, we demonstrate that previous results obtained for neural networks hold for models without regularization terms or large sample sizes. We show how these results are used to adjust the multiple correlation coefficient by the degrees of freedom. The methodology is demonstrated using simulated data from a Cobb-Douglas type function.