Hyperparameter Optimization: Comparing Genetic Algorithm against Grid Search and Bayesian Optimization

The performance of machine learning algorithms are affected by several factors, some of these factors are related to data quantity, quality, or its features. Another element is the choice of an appropriate algorithm to solve the problem and one major influence is the parameter configuration based on the problem specification. Parameters in machine learning can be classified in two types: (1) model parameters that are internal, configurable, and its value can be estimated from data such as weights of a deep neural network; and (2) hyperparameters, which are external and its values can not be estimated from data such as the learning rate for the training of a neural network. Hyperparameter values may be specified by a practitioner or using a heuristic, or parameter values obtained from other problems can be used etc., however, the best values of these parameters are identified when the algorithm has the highest accuracy, and these could be achieved by tuning the parameters. The main goal of this paper is to conduct a comparison study between different algorithms that are used in the optimization process in order to find the best hyperparameter values for the neural network. The algorithms applied are grid search algorithm, bayesian algorithm, and genetic algorithm. Different evaluation measures are used to conduct this comparison such as accuracy and running time.