Hyper-Parameter Optimization for Deep Learning by Surrogate-based Model with Weighted Distance Exploration

To improve deep neural net hyper-parameter optimization we develop a deterministic surrogate optimization algorithm as an efficient alternative to Bayesian optimization. A deterministic Radial Basis Function (RBF) surrogate model is built to interpolate previously evaluated points, and this surrogate model is incrementally updated in each iteration. The stochastic algorithm CMA-ES is used to search the acquisition function based on the surrogate. The acquisition function at a point is based on a weighted average of the surrogate at x and the minimum distance from x to a previously evaluated point. We evaluate the proposed algorithm RBF-CMA on hyper-parameter optimization tasks for deep convolutional neural networks on datasets of CIFAR-10, SVHN, and CIFAR-100. We show that RBF-CMA achieves a promising performance especially when the search space dimension is high in comparison to other algorithms including GP-EI, GP-LCB, and SMBO.