Detecting Gender as a Moderator in Meta-Analysis: The Problem of Restricted Between-Study Variance

Meta-analyses in the psychological sciences typically examine moderators that may explain heterogeneity in effect sizes. One of the most commonly examined moderators is gender. Overall, tests of gender as a moderator are rarely significant, which may be because effects rarely differ substantially between men and women. While this may be true in some cases, we also suggest that the lack of significant findings may be attributable to the way in which gender is examined as a meta-analytic moderator, such that detecting moderating effects is very unlikely even when such effects are substantial in magnitude. More specifically, we suggest that lack of between-primary study variance in gender composition makes it exceedingly difficult to detect moderation. That is, because primary studies tend to have similar male-to-female ratios, there is very little variance in gender composition between primaries, making it nearly impossible to detect between-study differences in the relationship of interest as a function of gender. In the present article, we report results from two studies: (a) a meta-meta-analysis in which we demonstrate the magnitude of this problem by computing the between-study variance in gender composition across 286 meta-analytic moderation tests from 50 meta-analyses, and (b) a Monte Carlo simulation study in which we show that this lack of variance results in near-zero moderator effects even when male-female differences in correlations are quite large. Our simulations are also used to show the value of single-gender studies for detecting moderating effects.