Controlling Decision Errors With Minimal Costs: The Sequential Probability Ratio t Test

For several years, the public debate in psychological science has been dominated by what is referred to as the reproducibility crisis. This crisis has, inter alia, drawn attention to the need for proper control of statistical decision errors in testing psychological hypotheses. However, conventional methods of error probability control often require fairly large samples. Sequential statistical tests provide an attractive alternative: They can be applied repeatedly during the sampling process and terminate whenever there is sufficient evidence in the data for one of the hypotheses of interest. Thus, sequential tests may substantially reduce the required sample size without compromising predefined error probabilities. Herein, we discuss the most efficient sequential design, the sequential probability ratio test (SPRT), and show how it is easily implemented for a 2-sample t test using standard statistical software. We demonstrate, by means of simulations, that the SPRT not only reliably controls error probabilities but also typically requires substantially smaller samples than standard t tests and other common sequential designs. Moreover, we investigate the robustness of the SPRT against violations of its assumptions. Finally, we illustrate the sequential t test by applying it to an empirical example and provide recommendations on how psychologists can employ it in their own research to benefit from its desirable properties. Translational Abstract Fostered by a series of unsuccessful attempts to replicate seemingly well-established empirical results, the reproducibility crisis has dominated the public debate in psychological science for several years. Apart from increasing awareness for the consequences of questionable research practices, the crisis has drawn attention to the shortcomings of currently dominating statistical procedures. Critically, conventional methods that allow for control of both Type I and Type II statistical error probabilities-alpha and beta, respectively- often require sample sizes much larger than typically employed. Therefore, we promote an alternative that requires substantially smaller sample sizes on average while still controlling error probabilities: sequential analysis. Unlike conventional tests, sequential tests are designed to be applied repeatedly during the sampling process and terminate as soon as there is sufficient evidence for one of the hypotheses of interest. Herein, we discuss the most efficient sequential design, the sequential probability ratio test (SPRT), and show how it is easily implemented for the common t test to compare means of 2 independent groups. We demonstrate by means of simulations that the SPRT reliably controls error probabilities and requires smaller samples than standard t tests or other common sequential designs. Moreover, we investigate the robustness of the SPRT against violations of its assumptions. Finally, we illustrate the sequential t test by applying it to an empirical example and provide concrete recommendations on how psychologists can employ it in their own research to benefit from its desirable properties.