Methodology in meta-analysis: A study from Critical Care meta-analytic practice

Methodological aspects of meta-analytic practice, heterogeneity, publication bias, metaregression and effect metric, were investigated in 14 meta-analyses reflecting major therapeutic concern in Critical Care practice. Compared with the standard Q test, the exact Zelen test was more sensitive in identifying heterogeneity. Assessment of heterogeneity impact by the I2 statistic was consistent with inferences afforded by both the Q and Zelen test. Publication bias was subject to test and metric determination: funnel plots exhibited variable asymmetry across studies and between metrics; the regression asymmetry test appeared more sensitive than the rank correlation test; the "trim and fill" method was the most sensitive, but suggested, on the basis of quantification of the effects of potentially missing studies, that meta-analyses may be resistant to such missingness. Metaregression of treatment effect against control risk using Bayesian hierarchical regression in all metrics (log odds ratio, log risk ratio and RD) suggested that naïve linear regression approaches over-diagnosed significant relationships and exhibited regression dilution. Heterogeneity, publication bias and risk related treatment effects all demonstrate estimator and metric dependence; the RD metric would appear the most capricious in this regard. © Springer Science + Business Media, LLC 2006.