Assessment of Study Quality in Psychological Meta-Analyses: A Systematic Review

Meta-analyses have developed into a preferred method for aggregating effect sizes reported in primary studies and are a viable toot for both researchers as well as political and social decision-makers. However, the meta-analytic estimates may be biased because of differences in the methodological rigor between primary studies. Accordingly, the assessment of primary study quality as well as the provision of appropriate quality appraisal tools should be an essential part of any scientific discipline that uses meta-analyses for the quantitative integration of study results. The present review provides an overview of whether and which quality criteria are used in psychological meta-analyses. A total of 225 meta-analyses published in Psychological Bulletin over the last 10 years were included in the analysis. Only about 18% of these studies explicitly consider the quality of the primary studies. Additionally, the specific approach as well as the instruments or items used show a high degree of heterogeneity between the studies. These differences, the advantages and disadvantages of the specific procedures as well as the practical implications arising from them are thoroughly discussed.