Model-based ideal testing of hardware description language (HDL) programs

An ideal test is supposed to show not only the presence of bugs but also their absence. Based on the Fundamental Test Theory of Goodenough and Gerhart (IEEE Trans Softw Eng SE-1(2):156–173, 1975), this paper proposes an approach to model-based ideal testing of hardware description language (HDL) programs based on their behavioral model. Test sequences are generated from both original (fault-free) and mutant (faulty) models in the sense of positive and negative testing, forming a holistic test view. These test sequences are then executed on original (fault-free) and mutant (faulty) HDL programs, in the sense of mutation testing. Using the techniques known from automata theory, test selection criteria are developed and formally show that they fulfill the major requirements of Fundamental Test Theory, that is, reliability and validity. The current paper comprises a preparation step (consisting of the sub-steps model construction, model mutation, model conversion, and test generation) and a composition step (consisting of the sub-steps pre-selection and construction of Ideal test suites). All the steps are supported by a toolchain that is already implemented and is available online. To critically validate the proposed approach, three case studies (a sequence detector, a traffic light controller, and a RISC-V processor) are used and the strengths and weaknesses of the approach are discussed. The proposed approach achieves the highest mutation score in positive and negative testing for all case studies in comparison with two existing methods (regular expression-based test generation and context-based random test generation), using four different techniques.