Applying Design of Experiments in Testing and Validation of Statistical Software

Validating statistical software involves a variety of challenges. Of these, the most difficult is the selection of an effective set of test cases, sometimes referred to as the "test case selection problem". To further complicate matters, for many statistical applications, development and validation are done by individuals who often have limited time to validate their application and may not have formal training in software validation techniques. As a result, it is imperative that the adopted validation method is efficient, as well as effective, and it should also be one that can be easily understood by individuals not trained in software validation techniques. As it turns out, the test case selection problem can be thought of as a design of experiments (DOE) problem. In this paper, we discuss how familiar DOE principles can be applied to testing and validating statistical software. We also briefly describe examples of how we have applied DOE principles in the testing and validation of statistical software, including how we addressed some of the challenges that are specific to that type of software.