ETNA: An Evaluation Platform for Property-Based Testing (Experience Report)

被引：3

作者：

Shi, Jessica ^{[1
]}

Keles, Alperen ^{[2
]}

Goldstein, Harrison ^{[1
]}

Pierce, Benjamin C. ^{[1
]}

Lampropoulos, Leonidas ^{[2
]}

机构：

[1] Univ Penn, Philadelphia, PA 19104 USA

[2] Univ Maryland, College Pk, MD 20742 USA

来源：

PROCEEDINGS OF THE ACM ON PROGRAMMING LANGUAGES-PACMPL | 2023年 / 7卷 / ICFP期

关键词：

property-based testing; empirical evaluation; mutation testing;

D O I：

10.1145/3607860

中图分类号：

TP31 [计算机软件];

学科分类号：

081202 ; 0835 ;

摘要：

Property-based testing is a mainstay of functional programming, boasting a rich literature, an enthusiastic user community, and an abundance of tools - so many, indeed, that new users may have difficulty choosing. Moreover, any given framework may support a variety of strategies for generating test inputs; even experienced users may wonder which are better in a given situation. Sadly, the PBT literature, though long on creativity, is short on rigorous comparisons to help answer such questions. We present Etna, a platform for empirical evaluation and comparison of PBT techniques. Etna incorporates a number of popular PBT frameworks and testing workloads from the literature, and its extensible architecture makes adding new ones easy, while handling the technical drudgery of performance measurement. To illustrate its benefits, we use Etna to carry out several experiments with popular PBT approaches in both Coq and Haskell, allowing users to more clearly understand best practices and tradeoffs.

引用

页数：17

共 53 条

[21] An Analysis and Survey of the Development of Mutation Testing [J].

Jia, Yue ;

Harman, Mark .

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 2011, 37 (05) :649-678

[22] Evaluating Fuzz Testing [J].

Klees, George ;

Ruef, Andrew ;

Cooper, Benji ;

Wei, Shiyi ;

Hicks, Michael .

PROCEEDINGS OF THE 2018 ACM SIGSAC CONFERENCE ON COMPUTER AND COMMUNICATIONS SECURITY (CCS'18), 2018, :2123-2138

[23]

Klein Casey, 2009, WORKSHOP SCHEME FUNC

[24]

Kuraj I., 2014, P 5 ANN SCALA WORKSH, P45, DOI 10.1145/2637647.2637655

[25]

Kuraj I, 2015, ACM SIGPLAN NOTICES, V50, P37, DOI [10.1145/2814270.2814323, 10.1145/2858965.2814323]

[26] Coverage Guided, Property Based Testing [J].

Lampropoulos, Leonidas ;

Hicks, Michael ;

Pierce, Benjamin C. .

PROCEEDINGS OF THE ACM ON PROGRAMMING LANGUAGES-PACMPL, 2019, 3 (OOPSLA)

[27] Generating Good Generators for Inductive Relations [J].

Lampropoulos, Leonidas ;

Paraskevopoulou, Zoe ;

Pierce, Benjamin C. .

PROCEEDINGS OF THE ACM ON PROGRAMMING LANGUAGES-PACMPL, 2018, 2 (POPL)

[28] Beginner's Luck A Language for Property-Based Generators [J].

Lampropoulos, Leonidas ;

Gallois-Wong, Diane ;

Hritcu, Catalin ;

Hughes, John ;

Pierce, Benjamin C. ;

Xia, Li-yao .

ACM SIGPLAN NOTICES, 2017, 52 (01) :114-129

[29]

Lampropoulos Leonidas, 2018, QuickChick: Property-Based Testing In Coq

[30]

Lampropoulos Leonidas, 2018, Ph. D. Dissertation

← 1 2 3 4 5 6 →