Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

被引:53
作者
Tendal, Britta [1 ]
Higgins, Julian P. T. [4 ]
Juni, Peter [2 ,3 ]
Hrobjartsson, Asbjorn [1 ]
Trelle, Sven [2 ,3 ]
Nuesch, Eveline [2 ,3 ]
Wandel, Simon [2 ,3 ]
Jorgensen, Anders W. [1 ]
Gesser, Katarina [5 ]
Ilsoe-Kristensen, Soren [5 ]
Gotzsche, Peter C. [1 ]
机构
[1] Rigshosp, Nord Cochrane Ctr, Dept 3343, DK-2100 Copenhagen, Denmark
[2] Univ Bern, Inst Social & Prevent Med, CH-3012 Bern, Switzerland
[3] Univ Hosp Bern, CTU Bern, Bern, Switzerland
[4] Univ Cambridge, Inst Publ Hlth, MRC, Biostat Unit, Cambridge CB2 1TN, England
[5] Univ Copenhagen, Fac Pharmaceut Sci, DK-1168 Copenhagen, Denmark
来源
BMJ-BRITISH MEDICAL JOURNAL | 2009年 / 339卷
关键词
COMPLIANCE THERAPY; DATA EXTRACTION; RANDOMIZED-TRIALS; ERRORS;
D O I
10.1136/bmj.b3128
中图分类号
R5 [内科学];
学科分类号
1002 ; 100201 ;
摘要
Objective To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. Design Observer agreement study. Data sources A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. Data extraction Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways ( 45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. Results The agreement was 53% at trial level and 31% at meta- analysis level. Including all pairs, the median disagreement was SMD=0.22 ( interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta- analysis level, individual observers reached different conclusions than the originally published review. Conclusions Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.
引用
收藏
页码:551 / 553
页数:7
相关论文
共 24 条
[1]   Fluticasone versus placebo for chronic asthma in adults and children [J].
Adams, N. P. ;
Bestall, J. C. ;
Lasserson, T. J. ;
Jones, P. W. ;
Cates, C. .
COCHRANE DATABASE OF SYSTEMATIC REVIEWS, 2005, (04)
[2]   Regional versus general anaesthesia for caesarean section [J].
Afolabi, B. B. ;
Lesi, F. E. A. ;
Merah, N. A. .
COCHRANE DATABASE OF SYSTEMATIC REVIEWS, 2006, (04)
[3]  
BUCKLEY LA, 2007, COCHRANE DATABASE SY
[4]   Single data extraction generated more errors than double data extraction in systematic reviews [J].
Buscemi, Nina ;
Harding, Lisa ;
Vandermeer, Ben ;
Tjosvold, Lisa ;
Klassen, Terry P. .
JOURNAL OF CLINICAL EPIDEMIOLOGY, 2006, 59 (07) :697-703
[5]   Empirical evidence for selective reporting of outcomes in randomized trials -: Comparison of Protocols to published articles [J].
Chan, AW ;
Hróbjartsson, A ;
Haahr, MT ;
Gotzsche, PC ;
Altman, DG .
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION, 2004, 291 (20) :2457-2465
[6]   Tayside-Fife clinical trial of cognitive-behavioural therapy for medication-resistant psychotic symptoms - Results to 3-month follow-up [J].
Durham, RC ;
Guthrie, M ;
Morton, RV ;
Reid, DA ;
Treliving, LR ;
Fowler, D ;
Macdonald, RR .
BRITISH JOURNAL OF PSYCHIATRY, 2003, 182 :303-311
[7]   Psychological treatments versus treatment as usual for obsessive compulsive disorder (OCD) [J].
Gava, I ;
Barbui, C. ;
Aguglia, E. ;
Carlino, D. ;
Churchill, R. ;
De Vanna, M. ;
McGuire, H. F. .
COCHRANE DATABASE OF SYSTEMATIC REVIEWS, 2007, (02)
[8]   Data extraction errors in meta-analyses that use standardized mean differences [J].
Gotzsche, Peter C. ;
Hrobjartsson, Asbjorn ;
Maric, Katja ;
Tendal, Britta .
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION, 2007, 298 (04) :430-437
[9]   Cost-effectiveness evaluation of compliance therapy for people with psychosis [J].
Healey, A ;
Knapp, M ;
Astin, J ;
Beecham, J ;
Kemp, R ;
Kirov, G ;
David, A .
BRITISH JOURNAL OF PSYCHIATRY, 1998, 172 :420-424
[10]  
Higgins JPT, 2008, Cochrane Handbook for Systematic Reviews of Interventions, DOI DOI 10.1002/9780470712184