THE PERFORMANCE PROFILE: A MULTI-CRITERIA PERFORMANCE EVALUATION METHOD FOR TEST-BASED PROBLEMS

被引:4
作者
Jaskowski, Wojciech [1 ]
Liskowski, Pawel [1 ]
Szubert, Marcin [1 ]
Krawiec, Krzysztof [1 ]
机构
[1] Poznan Univ Tech, Inst Comp Sci, Ul Piotrowo 2, PL-60965 Poznan, Poland
关键词
coevolutionary algorithms; evolution strategies; Othello; Reversi; games; multi-objective analysis; EVOLUTION; COEVOLUTION; DIVERSITY; IMPROVE; CHOICES;
D O I
10.1515/amcs-2016-0015
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
In test-based problems, solutions produced by search algorithms are typically assessed using average outcomes of interactions with multiple tests. This aggregation leads to information loss, which can render different solutions apparently indifferent and hinder comparison of search algorithms. In this paper we introduce the performance profile, a generic, domain-independent, multi-criteria performance evaluation method that mitigates this problem by characterizing the performance of a solution by a vector of outcomes of interactions with tests of various difficulty. To demonstrate the usefulness of this gauge, we employ it to analyze the behavior of Othello and Iterated Prisoner's Dilemma players produced by five (co) evolutionary algorithms as well as players known from previous publications. Performance profiles reveal interesting differences between the players, which escape the attention of the scalar performance measure of the expected utility. In particular, they allow us to observe that evolution with random sampling produces players coping well against the mediocre opponents, while the coevolutionary and temporal difference learning strategies play better against the high-grade opponents. We postulate that performance profiles improve our understanding of characteristics of search algorithms applied to arbitrary test-based problems, and can prospectively help design better methods for interactive domains.
引用
收藏
页码:215 / 229
页数:15
相关论文
共 45 条
  • [1] [Anonymous], 2006, The Evolution of Cooperation
  • [2] Evolution strategies – A comprehensive introduction
    Hans-Georg Beyer
    Hans-Paul Schwefel
    [J]. Natural Computing, 2002, 1 (1) : 3 - 52
  • [3] Agent-Case Embeddings for the Analysis of Evolved Systems
    Ashlock, Daniel
    Lee, Colin
    [J]. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, 2013, 17 (02) : 227 - 240
  • [4] Bucci A, 2004, LECT NOTES COMPUT SC, V3102, P501
  • [5] Measuring generalization performance in coevolutionary learning
    Chong, Siang Yew
    Tino, Peter
    Yao, Xin
    [J]. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, 2008, 12 (04) : 479 - 505
  • [6] Improving Generalization Performance in Co-Evolutionary Learning
    Chong, Siang Yew
    Tino, Peter
    Ku, Day Chyi
    Yao, Xin
    [J]. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, 2012, 16 (01) : 70 - 85
  • [7] Relationship Between Generalization and Diversity in Coevolutionary Learning
    Chong, Siang Yew
    Tino, Peter
    Yao, Xin
    [J]. IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, 2009, 1 (03) : 214 - 232
  • [8] Behavioral diversity, choices and noise in the iterated prisoner's dilemma
    Chong, SY
    Yao, X
    [J]. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, 2005, 9 (06) : 540 - 551
  • [9] Darwen P. J., 2000, International Journal of Knowledge-Based Intelligent Engineering Systems, V4, P191
  • [10] Darwen PJ, 2001, IEEE C EVOL COMPUTAT, P987, DOI 10.1109/CEC.2001.934298