What is the probability of replicating a statistically significant effect?

被引:53
作者
Miller, Jeff [1 ]
机构
[1] Univ Otago, Dept Psychol, Dunedin, New Zealand
关键词
CONFIDENCE-INTERVALS; PSYCHOLOGICAL-RESEARCH; GAMBLERS FALLACY; HOT HAND; P VALUES; POWER; INFERENCE;
D O I
10.3758/PBR.16.4.617
中图分类号
B841 [心理学研究方法];
学科分类号
040201 ;
摘要
If an initial experiment produces a statistically significant effect, what is the probability that this effect will be replicated in a follow-up experiment? I argue that this seemingly fundamental question can be interpreted in two very different ways and that its answer is, in practice, virtually unknowable under either interpretation. Although the data from an initial experiment can be used to estimate one type of replication probability, this estimate will rarely be precise enough to be of any use. The other type of replication probability is also unknowable, because it depends on unknown aspects of the research context. Thus, although it would be nice to know the probability of replicating a significant effect, researchers must accept the fact that they generally cannot determine this information, whichever type of replication probability they seek.
引用
收藏
页码:617 / 640
页数:24
相关论文
共 50 条
[1]   Detecting Statistically Significant Communities [J].
He, Zengyou ;
Liang, Hao ;
Chen, Zheng ;
Zhao, Can ;
Liu, Yan .
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, 2022, 34 (06) :2711-2725
[2]   Errors in the interpretation of 'no statistically significant difference' [J].
Gibbs, N. M. .
ANAESTHESIA AND INTENSIVE CARE, 2013, 41 (02) :151-153
[3]   Estimating risk from underpowered, but statistically significant, studies: was APPROVe on TARGET? [J].
La Caze, A. ;
Duffull, S. .
JOURNAL OF CLINICAL PHARMACY AND THERAPEUTICS, 2011, 36 (06) :637-641
[4]   Coup de Grace for a Tough Old Bull: "Statistically Significant" Expires [J].
Hurlbert, Stuart H. ;
Levine, Richard A. ;
Utts, Jessica .
AMERICAN STATISTICIAN, 2019, 73 :352-357
[5]   HOW TO STATISTICALLY SHOW THE ABSENCE OF AN EFFECT [J].
Quertemont, Etienne .
PSYCHOLOGICA BELGICA, 2011, 51 (02) :109-127
[6]   Mining Statistically Significant Co-location and Segregation Patterns [J].
Barua, Sajib ;
Sander, Joerg .
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, 2014, 26 (05) :1185-1199
[7]   A simulation study of the strength of evidence in the recommendation of medications based on two trials with statistically significant results [J].
van Ravenzwaaij, Don ;
Ioannidis, John P. A. .
PLOS ONE, 2017, 12 (03)
[8]   Estimating the replicability of statistically significant moderation effects in personality research using z-curve analysis [J].
Sotola, Lukas K. ;
Crede, Marcus .
JOURNAL OF RESEARCH IN PERSONALITY, 2023, 107
[9]   Statistically significant? Let us recognize that estimates of tested effects are uncertain [J].
Bultez, Alain ;
Derbaix, Christian ;
Herrmann, Jean-Luc .
RECHERCHE ET APPLICATIONS EN MARKETING-ENGLISH EDITION, 2022, 37 (01) :82-105
[10]   SONAR: Do a New Design and Statistically Significant Results Translate to Reliability? [J].
Walsh, Michael .
CLINICAL JOURNAL OF THE AMERICAN SOCIETY OF NEPHROLOGY, 2020, 15 (06) :889-891