GECCO Black-Box Optimization Competitions: Progress from 2009 to 2018

The GECCO Workshop on Real-Parameter Black-Box Optimization Benchmarking Series is a series of benchmarking workshops held every year since 2009 that evaluates the performance of new optimization algorithms. Originally, the workshop organizers provided results for every year the workshop took place. In this article, we directly compare algorithms from every workshop held from 2009 to 2018. We analyze the compared algorithms on the noiseless benchmark function set using the competition's official empirical approach, and with a statistical approach called Deep Statistical Comparison. Our goal is to investigate how algorithm performance has evolved throughout the years, and to show differences between empirical and statistical approaches to evaluating results.