System Test Architecture Evaluation: A Probabilistic Modeling Approach

In this paper, we study the effects of testing architecture on system quality using a probabilistic model of unit testing. The Markovian view of the testing process relates unit quality and test quality to the expected unit/system quality, and to the expected number of tests. The model is based on replicating testing where the test is replicated only after a fail test outcome. A set of equations are generalized for many component systems. The study enables the costs and associated benefits for particular groups of components to be considered for module testing at different levels of system hierarchy. Simulation results show that the selection of an appropriate testing architecture and modular architecture can greatly enhance the efficiency and effectiveness of system testing. The model is applied to several testing architectures and test patterns to illustrate the tradeoff comparisons that can be made between different system architectures in terms of the overall test cost and the resulting system quality. Several heuristics are derived to assist in planning tests of complex systems for optimizing quality and cost before systems are built. This Markovian model of testability in systems shows promise for use from the modular unit build-level in engineering manufacture development through to strategizing portfolio-level integration and information assurance testing of new capability projects with legacy systems in a family-of-systems. The early abstract testability work reinforces the benefits of iterating testing as early and often as possible. Following trials to validate the model in different contexts and to refine the interface with systems engineers, this testability model could be added to other design for six-sigma tools and techniques to better enable systems engineering practitioners to tradeoff on testability options well before testing starts.