Background. Chinese herbal medicines reportedly increase efficacy and minimize toxicity of chemotherapy; however, little attention has been paid to how poor study quality can bias outcomes. Methods. We systematically searched MEDLINE, TCMLARS, EMBASE, and Cochrane Library for randomized controlled trials of Chinese herbal medicines combined with fluorouracil-based chemotherapy compared with the same chemotherapy alone. We screened for eligibility, extracted data, and pooled data with random-effects meta-analysis. Outcome measures were survival, toxicity, tumor response, performance status, quality of life, and Cochrane Risk of Bias (ROB) criteria to critically evaluate the quality of reporting in the randomized trials included in the meta-analysis. Results. We found 36 potentially eligible studies, with only 3 (those with low ROB) qualifying for meta-analysis. Two reported chemotherapy-related diarrhea reduced by 57% (relative risk [RR] = 0.43; 95% CI = 0.19-1.01; I-2 test for variation in RR due to heterogeneity = 0.0%), with nonsignificant results. Two reported white blood cell toxicity reduced by 66% (RR = 0.34; 95% CI = 0.16-0.72; I-2 test for variation in RR due to heterogeneity = 0.0%), with statistically significant results. Stratifying analysis by studies with high versus low ROB, we found substantial overestimation of benefit: Studies with high ROB overestimated by nearly 2-fold reduction of platelet toxicity by Chinese herbal medicines (RR = 0.35, 95% CI = 0.15-0.84 vs RR = 0.65, 95% CI = 0.11-3.92). Studies with high ROB overestimated by nearly 2-fold reduction of vomiting toxicity (RR = 0.45, 95% CI = 0.33-0.61 vs RR = 0.87, 95% CI = 0.48-1.58). And, studies with high ROB overestimated by 21% the reduction in diarrhea toxicity (RR = 0.34, 95% CI = 0.20-0.58 vs RR = 0.43, 95% CI = 0.19-1.01). Studies with high ROB also overestimated by 16% improvement in tumor response (RR = 1.39, 95% CI = 1.18-1.63 vs RR = 1.20; 95% CI = 0.81-1.79). Not accounting for ROB would have exaggerated evidence of benefit and failed to detect nonsignificance of results. Conclusions. In the present analysis, involving 36 studies, 2593 patients, 20 outcomes, 36 medical institutions, and 271 named research authors, 92% of the data points were from studies at high ROB. Given the poor quality of the data in studies identified, it cannot be concluded whether combining Chinese herbs with chemotherapy reduces toxicity of chemotherapy.
