Convergence Analysis of PSO for Hyper-Parameter Selection in Deep Neural Networks

Deep Neural Networks (DNNs) have gained enormous research attention since they consistently outperform other state-of-the-art methods in a plethora of machine learning tasks. However, their performance strongly depends on the DNN hyper-parameters which are commonly tuned by experienced practitioners. Recently, we introduced Particle Swarm Optimization (PSO) and parallel PSO techniques to automate this process. In this work, we theoretically and experimentally investigate the convergence capabilities of these algorithms. The experiments were performed for several DNN architectures (both gradually augmented and hand-crafted by a human) using two challenging multi-class benchmark datasets-MNIST and CIFAR-10.