Abstract—Reporting the results of optimization algorithms in evolutionary computation is a challenging task with many potential pitfalls. The source of problems is their stochastic nature and inability to guarantee an optimal solution in polynomial time. One of the basic questions that is often not addressed concerns the method of summarizing the entire distribution of solutions into a single value. Although the mean value is used by default for that purpose, the best solution obtained is also occasionally used in addition to or instead of it. Based on our analysis of different possibilities for measuring the performance of stochastic optimization algorithms presented in this paper we propose quantiles as the standard measure of performance. Quantiles can be naturally interpreted for the designated purpose. Besides, they are defined even when the arithmetic mean is not, and are applicable in cases of multiple executions of an algorithm. Our study also showed that, on the contrary to many other fields, in the case of stochastic optimization algorithms the greater variability in measured data can be considered as an advantage.
Index Terms—Algorithmic performance, experimental evaluation, metaheuristics, quantile.
Nikola Ivkovic is with the Faculty of Organization and Informatics, University of Zagreb, Pavlinska 2, HR 42000 Varazdin, Croatia (e-mail: nikola.ivkovic@foi.hr).
Domagoj Jakobovic and Marin Golub are with the Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, HR 10000 Zagreb, Croatia (e-mail: domagoj.jakobovic@fer.hr, marin.golub@fer.hr).
Cite: Nikola Ivkovic, Domagoj Jakobovic, and Marin Golub, "Measuring Performance of Optimization Algorithms in Evolutionary Computation," International Journal of Machine Learning and Computing vol. 6, no. 3, pp. 167-171, 2016.