Would wider adoption of reproducible research be beneficial for empirical software engineering research?

被引:28
作者
Madeyski, Lech [1 ]
Kitchenham, Barbara [2 ]
机构
[1] Wroclaw Univ Sci & Technol, Fac Comp Sci & Management, Wyb Wyspianskiego 27, PL-50370 Wroclaw, Poland
[2] Keele Univ, Sch Comp & Math, Keele, Staffs, England
关键词
Reproducible research; empirical software engineering; scientific practice;
D O I
10.3233/JIFS-169146
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Researchers have identified problems with the validity of software engineering research findings. In particular, it is often impossible to reproduce data analyses, due to lack of raw data, or sufficient summary statistics, or undefined analysis procedures. The aim of this paper is to raise awareness of the problems caused by unreproducible research in software engineering and to discuss the concept of reproducible research (RR) as a mechanism to address these problems. RR is the idea that the outcome of research is both a paper and its computational environment. We report some recent studies that have cast doubts on the reliability of research outcomes in software engineering. Then we discuss the use of RR as a means of addressing these problems. We discuss the use of RR in software engineering research and present the methodology we have used to adopt RR principles. We report a small working example of how to create reproducible research. We summarise advantages of and problems with adopting RR methods. We conclude that RR supports good scientific practice and would help to address some of the problems found in empirical software engineering research.
引用
收藏
页码:1509 / 1521
页数:13
相关论文
共 48 条
  • [21] Why most published research findings are false
    Ioannidis, JPA
    [J]. PLOS MEDICINE, 2005, 2 (08) : 696 - 701
  • [22] Incorrect results in software engineering experiments: How to improve research practices
    Jorgensen, Magne
    Dyba, Tore
    Liestol, Knut
    Sjoberg, Dag I. K.
    [J]. JOURNAL OF SYSTEMS AND SOFTWARE, 2016, 116 : 133 - 145
  • [23] Cross-Project Defect Prediction with Respect to Code Ownership Model: an Empirical Study
    Jureczko, Marian
    Madeyski, Lech
    [J]. E-INFORMATICA SOFTWARE ENGINEERING JOURNAL, 2015, 9 (01) : 21 - 35
  • [24] Kitchenham BA, 2016, EVIDENCE BASED SOFTW, P133
  • [25] Robust Statistical Methods for Empirical Software Engineering
    Kitchenham, Barbara
    Madeyski, Lech
    Budgen, David
    Keung, Jacky
    Brereton, Pearl
    Charters, Stuart
    Gibbs, Shirley
    Pohthong, Amnart
    [J]. EMPIRICAL SOFTWARE ENGINEERING, 2017, 22 (02) : 579 - 630
  • [26] LITERATE PROGRAMMING
    KNUTH, DE
    [J]. COMPUTER JOURNAL, 1984, 27 (02) : 97 - 111
  • [27] Kuhn M., CRAN Task View: Reproducible Research
  • [28] Leisch F., 2002, R News, V2, P28, DOI DOI 10.1159/000323281
  • [29] Madeyski L., 2016, REPRODUCER REPRODUCE
  • [30] Software Measurement and Defect Prediction with Depress Extensible Framework
    Madeyski, Lech
    Majchrzak, Marek
    [J]. FOUNDATIONS OF COMPUTING AND DECISION SCIENCES, 2014, 39 (04) : 249 - 270