Unsupervised Entropy-Based Selection of Data Sets for Improved Model Fitting

A method based on the information theory concept of entropy is presented for selecting subsets of data for off-line model identification. By using entropy-based data selection instead of random equiprobable sampling before training models, significant improvements are achieved in parameter convergence, accuracy and generalisation ability. Furthermore, model evaluation metrics exhibit less variance, therefore allowing faster convergence when multiple modelling trials have to be executed. These features are experimentally demonstrated by the results of an extensive number of neural network predictive modelling experiments, where the single difference in the identification of pairs of models was the data set used to tune model parameters. Unlike most active learning and instance selection procedures, the method is not iterative, does not rely on an existing model, and does not require a specific modelling technique. Instead, it selects data points in one unsupervised step relying solely on Shannon's information measure.