Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

被引：308

作者：

Cramer, Angelique O. J. ^{[1
]}

van Ravenzwaaij, Don ^{[2
]}

Matzke, Dora ^{[1
]}

Steingroever, Helen ^{[1
]}

Wetzels, Ruud ^{[3
]}

Grasman, Raoul P. P. P. ^{[1
]}

Waldorp, Lourens J. ^{[1
]}

Wagenmakers, Eric-Jan ^{[1
]}

机构：

[1] Univ Amsterdam, Dept Psychol, Psychol Methods, Amsterdam, Netherlands

[2] Univ Newcastle, Fac Sci & Informat Technol, Sch Psychol, Callaghan, NSW 2308, Australia

[3] Price Waterhouse Coopers, Data Analyt, Amsterdam, Netherlands

来源：

PSYCHONOMIC BULLETIN & REVIEW | 2016年 / 23卷 / 02期

关键词：

Multiway ANOVA; Factorial ANOVA; Multiple comparison problem; Type I error; Sequential Bonferroni; FALSE DISCOVERY RATE; BONFERRONI PROCEDURE; REGISTERED-REPORTS; STATISTICAL POWER; TESTS;

D O I：

10.3758/s13423-015-0913-5

中图分类号：

B841 [心理学研究方法];

学科分类号：

040201 ;

摘要：

Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

引用

页码：640 / 647

页数：8

共 41 条

[1] Registered Reports and Replications in Attention, Perception, & Psychophysics [J].

不详 .

ATTENTION PERCEPTION & PSYCHOPHYSICS, 2013, 75 (05) :781-783

[2]

[Anonymous], 1969, Methodology: Foundations of inference and research in the behavioral sciences

[3]

[Anonymous], 2004, The Journal of Socio-Economics, DOI [10.1016/j.socec.2004.09.033, DOI 10.1016/J.SOCEC.2004.09.033]

[4]

[Anonymous], 1994, The Collected Works of John W. Tukey

[5]

Barber T.X., 1976, Pitfalls in human research: Ten pivotal points

[6] Controlling the false discovery rate in behavior genetics research [J].

Benjamini, Y ;

Drai, D ;

Elmer, G ;

Kafkafi, N ;

Golani, I .

BEHAVIOURAL BRAIN RESEARCH, 2001, 125 (1-2) :279-284

[7]

Benjamini Y, 2001, ANN STAT, V29, P1165

[8] CONTROLLING THE FALSE DISCOVERY RATE - A PRACTICAL AND POWERFUL APPROACH TO MULTIPLE TESTING [J].

BENJAMINI, Y ;

HOCHBERG, Y .

JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES B-STATISTICAL METHODOLOGY, 1995, 57 (01) :289-300

[9] Power failure: why small sample size undermines the reliability of neuroscience [J].

Button, Katherine S. ;

Ioannidis, John P. A. ;

Mokrysz, Claire ;

Nosek, Brian A. ;

Flint, Jonathan ;

Robinson, Emma S. J. ;

Munafo, Marcus R. .

NATURE REVIEWS NEUROSCIENCE, 2013, 14 (05) :365-376

[10]

Chambers C., 2013, GUARDIAN

← 1 2 3 4 5 →