Multiple comparison procedures applied to model selection

This paper presents a new approach to model selection based on hypothesis testing. We first describe a procedure to generate different scores for any candidate model from a single sample of training data and then discuss how to apply multiple comparison procedures (MCP) to model selection. MCP statistical tests allow us to compare three or more groups of data while controlling the probability of making at least one Type I error. The complete procedure is illustrated on several model selection tasks, including the determination of the number of hidden units for feed-forward neural networks and the number of kernels for RBF networks. (C) 2002 Elsevier Science B.V. All rights reserved.