The problem with unadjusted multiple and sequential statistical testing

被引:36
作者
Albers, Casper [1 ]
机构
[1] Univ Groningen, Groningen, Netherlands
关键词
D O I
10.1038/s41467-019-09941-0
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
In research studies, the need for additional samples to obtain sufficient statistical power has often to be balanced with the experimental costs. One approach to this end is to sequentially collect data until you have sufficient measurements, e.g., when the p-value drops below 0.05. I outline that this approach is common, yet that unadjusted sequential sampling leads to severe statistical issues, such as an inflated rate of false positive findings. As a consequence, the results of such studies are untrustworthy. I identify the statistical methods that can be implemented in order to account for sequential sampling.
引用
收藏
页数:4
相关论文
共 27 条
[1]   Questionable research practices among italian research psychologists [J].
Agnoli, Franca ;
Wicherts, Jelte M. ;
Veldkamp, Coosje L. S. ;
Albiero, Paolo ;
Cubelli, Roberto .
PLOS ONE, 2017, 12 (03)
[2]  
Albers C.J., 2018, NIEUW ARCH WISKD, V5, P209
[3]   Adjust for Multiple Comparisons? It's Not That Simple [J].
Althouse, Andrew D. .
ANNALS OF THORACIC SURGERY, 2016, 101 (05) :1644-1645
[4]  
Altman DG., 1991, Practical statistics for medical research
[5]  
[Anonymous], 1985, Sequential Analysis
[6]  
Bartroff J., 2013, Sequential experimentation in clinical trials: Design and analysis
[7]   Adjusting for multiple testing - when and how? [J].
Bender, R ;
Lange, S .
JOURNAL OF CLINICAL EPIDEMIOLOGY, 2001, 54 (04) :343-349
[8]   Redefine statistical significance [J].
Benjamin, Daniel J. ;
Berger, James O. ;
Johannesson, Magnus ;
Nosek, Brian A. ;
Wagenmakers, E. -J. ;
Berk, Richard ;
Bollen, Kenneth A. ;
Brembs, Bjoern ;
Brown, Lawrence ;
Camerer, Colin ;
Cesarini, David ;
Chambers, Christopher D. ;
Clyde, Merlise ;
Cook, Thomas D. ;
De Boeck, Paul ;
Dienes, Zoltan ;
Dreber, Anna ;
Easwaran, Kenny ;
Efferson, Charles ;
Fehr, Ernst ;
Fidler, Fiona ;
Field, Andy P. ;
Forster, Malcolm ;
George, Edward I. ;
Gonzalez, Richard ;
Goodman, Steven ;
Green, Edwin ;
Green, Donald P. ;
Greenwald, Anthony ;
Hadfield, Jarrod D. ;
Hedges, Larry V. ;
Held, Leonhard ;
Ho, Teck Hua ;
Hoijtink, Herbert ;
Hruschka, Daniel J. ;
Imai, Kosuke ;
Imbens, Guido ;
Ioannidis, John P. A. ;
Jeon, Minjeong ;
Jones, James Holland ;
Kirchler, Michael ;
Laibson, David ;
List, John ;
Little, Roderick ;
Lupia, Arthur ;
Machery, Edouard ;
Maxwell, Scott E. ;
McCarthy, Michael ;
Moore, Don ;
Morgan, Stephen L. .
NATURE HUMAN BEHAVIOUR, 2018, 2 (01) :6-10
[9]   Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies [J].
Cramer, Angelique O. J. ;
van Ravenzwaaij, Don ;
Matzke, Dora ;
Steingroever, Helen ;
Wetzels, Ruud ;
Grasman, Raoul P. P. P. ;
Waldorp, Lourens J. ;
Wagenmakers, Eric-Jan .
PSYCHONOMIC BULLETIN & REVIEW, 2016, 23 (02) :640-647
[10]   Questionable Research Practices Revisited [J].
Fiedler, Klaus ;
Schwarz, Norbert .
SOCIAL PSYCHOLOGICAL AND PERSONALITY SCIENCE, 2016, 7 (01) :45-52