Bootstrap-Based Between-Study Heterogeneity Tests in Meta-Analysis

Meta-analysis combines pertinent information from existing studies to provide an overall estimate of population parameters/effect sizes, as well as to quantify and explain the differences between studies. However, testing between-study heterogeneity is one of the most challenging tasks in meta-analysis research. Existing methods for testing heterogeneity, such as the Q test and likelihood ratio (LR) test, have been criticized for their failure to control Type I error rate and/or failure to attain enough statistical power. Although better reference distribution approximations have been proposed in the literature, their application is limited. Additionally, when the interest is to test whether the size of the heterogeneity is larger than a specific level, existing methods are far from mature. To address these issues, we propose new heterogeneity tests. Specifically, we combine bootstrap methods with existing heterogeneity tests (i.e., the maximum LR test, the restricted maximum LR test, and the Q test) to overcome the reference distribution issue and denote them as B-ML-LRT, B-REML-LRT, and B-Q, respectively. Simulation studies were conducted to examine and compare the performance of the proposed methods with the regular LR test, the regular Q test, and the Kulinskaya's improved Q test in both random- and mixed-effects meta-analyses. Based on the results of Type I error rates and statistical power, B-REML-LRT is recommended. Additionally, the improved Q test is also recommended when it is applicable. An R package boot.heterogeneity is provided to facilitate the implementation of the proposed tests.