A genetic mixed-integer optimization of neural network hyper-parameters

Neural networks (NN) have become immensely popular for their effectiveness and flexibility in learning complicated patterns. Despite this success, they are often considered difficult to design because of the wide variety of parameters they require. Determining the most optimal selection of parameters has become tedious and costly, and neural architecture search (NAS) methods have been employed to try and take the guesswork out of the process. A common NAS approach is the genetic algorithm (GA); however, its usage is often exclusively tied to either the learnable parameters, or the meta-parameters that augment the learning. This work proposes an experimental approach for optimizing both real-valued weights and discrete meta-parameters simultaneously. Experimental results have shown that the current approach evolves both parameter sets effectively for simple problems like Iris, but still struggles in finding an optimal model for more rigorous problems.