A summary-attainment-surface plotting method for visualizing the performance of stochastic multiobjective optimizers

When evaluating the performance of a stochastic optimizer it is sometimes desirable to express peformance in terms of the quality attained in a certain fraction of sample runs. For example, the sample median quality is the best estimator of what one would expect to achieve in 50% of runs, and similarly for other quantiles. In multiobjective optimization, the notion still applies but the outcome of a run is measured not as a scalar (i.e. the cost of the best solution), but as an attainment surface in k-dimensional space (where k is the number of objectives). In this paper we report an algorithm that can be conveniently used to plot summary attainment surfaces in any number of dimensions (though it is particularly suited for three). A summary attainment surface is defined as the union of all tightest goals that have been attained (independently) in precisely s of the runs of a sample of n runs, for any s epsilon 1..n, and for any k. We also discuss the computational complexity of the algorithm and give some examples of its use. C code for the algorithm is available from the author.