An Analysis of Software Engineering Experiments Using Human Subjects

Content: Researchers perform experiments to check their proposals under controlled conditions. Thus, experiments are an important category of empirical studies and are the classical approach for identifying cause-effect relationships. Goal: Quantitatively characterize and analyze the controlled experiments in software engineering published in journal and conference proceedings over the decade 2003-2013. Method: We performed a systematic mapping study that includes all full papers published at EASE, ESEM and ESEJ. Results: We obtained 110 papers that report controlled experiments. In these experiments we obtained quantitative data about authors and institutions, subjects, tasks, environment, replication and threats to validity. Conclusions: The main contribution of this work is to show the amount of experiments published in the three main venues of Empirical Software Engineering between the years 2003-2013 and also how these experiments have being reported and executed.