Dicing with chance, life and death in systematic reviews and meta-analyses: DICE 3, a simulation study

Objectives: To show the effects of chance on meta-analyses, and the potential dangers of being prompted to do a meta-analysis by one favourable trial. Design: In total, 100,000 trials were simulated and combined into 10,000 meta-analyses, using data from the control group of a cancer trial. Each participant record was randomly coded to simulate allocation to 'treatment' or 'control'. Setting: Simulated study. Participants: De-identified records for 578 patients from the control group of a cancer trial, of whom 147 had died. Main outcome measure: Time to death from any cause. Results: Of the 100,000 trials, 4897 (4.9%) were statistically significant at 2p < 0.05 and 123 (1.2%) of the 10,000 meta-analyses were significant at 2p < 0.01. The most extreme result was a 20% reduction (99% CI: 0.70-0.91; 2p = 0.00002) in the annual odds of dying in the 'treatment' group. If a meta-analysis contained at least one trial with a statistically significant result (at 2p < 0.05), the likelihood of the meta-analysis being significant (at 2p < 0.01) increased strikingly. For example, among the 473 meta-analyses in which the first trial in a batch of 10 was statistically significant (at 2p < 0.05), 18 (3.8%) favoured treatment at 2p < 0.01. Conclusions: Chance can influence the results of meta-analyses regardless of how well they are conducted. Researchers should not ignore this when they plan a meta-analysis and when they report their results. People reading their reports should also be wary. Caution is particularly important when the results of one or more included studies influenced the decision to do the meta-analysis.