Test Case Prioritization Technique in Continuous Integration Based on Reinforcement Learning

As software delivery increasingly emphasizes speed and reliability, continuous integration (CI) has attracted more and more attention these years. Developers continue to integrate working copies into the mainline to realize software evolution. Each integration involves automated tests to verify whether the update introduces faults. However, as the scale of software increases, test suites contain more and more test cases. As software evolves, the coverage and fault detection ability of test cases also change among different CI cycles. As a result, the traditional test case prioritization techniques may be inapplicable. Techniques based on reinforcement learning can adjust prioritization strategies dynamically according to test feedback. But the existing techniques based on reinforcement learning proposed in recent years do not comprehensively consider information in the test suite during prioritization, which limits their effectiveness. This study proposes a new test case prioritization method in CI, called pointer ranking method. The method uses features like history information of test cases as inputs. In each CI cycle, the agent uses the attention mechanism to gain attention to all candidate test cases, and then obtains a prioritization result. After test execution, it obtains the updating direction from the feedback. It constantly adjusts its prioritization strategy in the process “prioritization, test execution, test feedback” and finally achieves satisfied prioritization performance. This study verifies the effectiveness of the proposed method on five large-scale datasets, and explores the impact of history length on method performance. Besides, it explores the model’s effectiveness on datasets which only contain regression test cases and the model’s execution efficiency. Finally, the study comes to the following conclusions. First, compared to existing techniques, pointer ranking method can adjust its strategy along with the evolution of the software, and effectively enhance the fault detection ability of test sequence in CI. Second, pointer ranking method has good robustness to history length. A small amount of history information can make it achieve the optimal performance. Third, pointer ranking method can handle regression test cases and newly-added test cases well. Finally, pointer ranking method has little time overhead. Considering its better and more stable prioritization performance, pointer ranking method is a very competitive method. © 2023 Chinese Academy of Sciences. All rights reserved.