Impact of test-driven development on productivity, code and tests: A controlled experiment

Context: Test-driven development is an approach to software development, where automated tests are written before production code in highly iterative cycles. Test-driven development attracts attention as well as followers in professional environment; however empirical evidence of its superiority regarding its effect on productivity, code and tests compared to test-last development is still fairly limited. Moreover, it is not clear if the supposed benefits come from writing tests before code or maybe from high iterativity/short development cycles. Objective: This paper describes a family of controlled experiments comparing test-driven development to micro iterative test-last development with emphasis on productivity, code properties (external quality and complexity) and tests (code coverage and fault-finding capabilities). Method: Subjects were randomly assigned to test-driven and test-last groups. Controlled experiments were conducted for two years, in an academic environment and in different developer contexts (pair programming and individual programming contexts). Number of successfully implemented stories, percentage of successful acceptance tests, McCabe's code complexity, code coverage and mutation score indicator were measured. Results: Experimental results and their selective meta-analysis show no statistically significant differences between test-driven development and iterative test-last development regarding productivity (chi(2)(6) = 4.799, p = 1.0, r = .107, 95% Cl (confidence interval): -.149 to .349), code complexity (chi(2)(6) = 8.094, p = .46, r = .048, 95% Cl: -.254 to .341), branch coverage (chi(2)(6) = 13.996, p = .059, r = .182, 95% Cl: -.081 to .421), percentage of acceptance tests passed (one experiment, Mann-Whitney U = 125.0, p = .98, r = .066) and mutation score indicator (chi(2)(4) = 3.807, p = .87, r = .128, 95% Cl: -.162 to .398). Conclusion: According to our findings, the benefits of test-driven development compared to iterative test-last development are small and thus in practice relatively unimportant, although effects are positive. There is an indication of test-driven development endorsing better branch coverage, but effect size is considered small. (C) 2011 Elsevier B.V. All rights reserved.