Replicability, Robustness, and Reproducibility in Psychological Science

被引:335
作者
Nosek, Brian A. [1 ,2 ]
Hardwicke, Tom E. [3 ]
Moshontz, Hannah [4 ]
Allard, Aurelien [5 ]
Corker, Katherine S. [6 ]
Dreber, Anna [7 ]
Fidler, Fiona [8 ]
Hilgard, Joe [9 ]
Struhl, Melissa Kline [2 ]
Nuijten, Michele B. [10 ]
Rohrer, Julia M. [11 ]
Romero, Felipe [12 ]
Scheel, Anne M. [13 ]
Scherer, Laura D. [14 ]
Schoenbrodt, Felix D. [15 ]
Vazire, Simine [16 ]
机构
[1] Univ Virginia, Dept Psychol, Charlottesville, VA 22904 USA
[2] Ctr Open Sci, Charlottesville, VA 22903 USA
[3] Univ Amsterdam, Dept Psychol, NL-1012 ZA Amsterdam, Netherlands
[4] Univ Wisconsin, Addict Res Ctr, Madison, WI 53706 USA
[5] Univ Calif Davis, Dept Psychol, Davis, CA 95616 USA
[6] Grand Valley State Univ, Psychol Dept, Allendale, MI 49401 USA
[7] Stockholm Sch Econ, Dept Econ, S-11383 Stockholm, Sweden
[8] Univ Melbourne, Sch Biosci, Parkville, Vic 3010, Australia
[9] Illinois State Univ, Dept Psychol, Normal, IL 61790 USA
[10] Tilburg Univ, Metares Ctr, NL-5037 AB Tilburg, Netherlands
[11] Univ Leipzig, Dept Psychol, D-04109 Leipzig, Germany
[12] Univ Groningen, Dept Theoret Philosophy, NL-9712 CP Groningen, Netherlands
[13] Eindhoven Univ Technol, Dept Ind Engn & Innovat Sci, NL-5612 AZ Eindhoven, Netherlands
[14] Univ Colorado, Anschutz Med Campus, Aurora, CO 80045 USA
[15] Ludwig Maximilian Univ Munich, Dept Psychol, D-80539 Munich, Germany
[16] Univ Melbourne, Sch Psychol Sci, Parkville, Vic 3052, Australia
基金
澳大利亚研究理事会; 荷兰研究理事会; 美国国家卫生研究院; 美国医疗保健研究与质量局;
关键词
replication; reproducibility; robustness; generalizability; research methods; statistical inference; validity; theory; metascience; REGISTERED REPLICATION REPORT; PUBLICATION DECISIONS; STATISTICAL POWER; SOCIAL-PSYCHOLOGY; EFFECT SIZE; TESTS; BIAS; CONSEQUENCES; TRANSPARENT; CREDIBILITY;
D O I
10.1146/annurev-psych-020821-114157
中图分类号
B84 [心理学];
学科分类号
04 ; 0402 ;
摘要
Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.
引用
收藏
页码:719 / 748
页数:30
相关论文
共 201 条
  • [1] Estimating the reproducibility of psychological science
    Aarts, Alexander A.
    Anderson, Joanna E.
    Anderson, Christopher J.
    Attridge, Peter R.
    Attwood, Angela
    Axt, Jordan
    Babel, Molly
    Bahnik, Stepan
    Baranski, Erica
    Barnett-Cowan, Michael
    Bartmess, Elizabeth
    Beer, Jennifer
    Bell, Raoul
    Bentley, Heather
    Beyan, Leah
    Binion, Grace
    Borsboom, Denny
    Bosch, Annick
    Bosco, Frank A.
    Bowman, Sara D.
    Brandt, Mark J.
    Braswell, Erin
    Brohmer, Hilmar
    Brown, Benjamin T.
    Brown, Kristina
    Bruening, Jovita
    Calhoun-Sauls, Ann
    Callahan, Shannon P.
    Chagnon, Elizabeth
    Chandler, Jesse
    Chartier, Christopher R.
    Cheung, Felix
    Christopherson, Cody D.
    Cillessen, Linda
    Clay, Russ
    Cleary, Hayley
    Cloud, Mark D.
    Cohn, Michael
    Cohoon, Johanna
    Columbus, Simon
    Cordes, Andreas
    Costantini, Giulio
    Alvarez, Leslie D. Cramblet
    Cremata, Ed
    Crusius, Jan
    DeCoster, Jamie
    DeGaetano, Michelle A.
    Della Penna, Nicolas
    den Bezemer, Bobby
    Deserno, Marie K.
    [J]. SCIENCE, 2015, 349 (6251)
  • [2] Registered Replication Report: Schooler and Engstler-Schooler (1990)
    Alogna, V. K.
    Attaya, M. K.
    Aucoin, P.
    Bahnik, S.
    Birch, S.
    Birt, A. R.
    Bornstein, B. H.
    Bouwmeester, S.
    Brandimonte, M. A.
    Brown, C.
    Buswell, K.
    Carlson, C.
    Carlson, M.
    Chu, S.
    Cislak, A.
    Colarusso, M.
    Colloff, M. F.
    Dellapaolera, K. S.
    Delvenne, J.-F.
    Di Domenico, A.
    Drummond, A.
    Echterhoff, G.
    Edlund, J. E.
    Eggleston, C. M.
    Fairfield, B.
    Franco, G.
    Gabbert, F.
    Gamblin, B. W.
    Garry, M.
    Gentry, R.
    Gilbert, E. A.
    Greenberg, D. L.
    Halberstadt, J.
    Hall, L.
    Hancock, P. J. B.
    Hirsch, D.
    Holt, G.
    Jackson, J. C.
    Jong, J.
    Kehn, A.
    Koch, C.
    Kopietz, R.
    Koerner, U.
    Kunar, M. A.
    Lai, C. K.
    Langton, S. R. H.
    Leite, F. P.
    Mammarella, N.
    Marsh, J. E.
    McConnaughy, K. A.
    [J]. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE, 2014, 9 (05) : 556 - 578
  • [3] Predicting the replicability of social science lab experiments
    Altmejd, Adam
    Dreber, Anna
    Forsell, Eskil
    Huber, Juergen
    Imai, Taisuke
    Johannesson, Magnus
    Kirchler, Michael
    Nave, Gideon
    Camerer, Colin
    [J]. PLOS ONE, 2019, 14 (12):
  • [4] Response to Comment on "Estimating the reproducibility of psychological science"
    Anderson, Christopher J.
    Bahnik, Stepan
    Barnett-Cowan, Michael
    Bosco, Frank A.
    Chandler, Jesse
    Chartier, Christopher R.
    Cheung, Felix
    Christopherson, Cody D.
    Cordes, Andreas
    Cremata, Edward J.
    Della Penna, Nicolas
    Estel, Vivien
    Fedor, Anna
    Fitneva, Stanka A.
    Frank, Michael C.
    Grange, James A.
    Hartshorne, Joshua K.
    Hasselman, Fred
    Henninger, Felix
    van der Hulst, Marije
    Jonas, Kai J.
    Lai, Calvin K.
    Levitan, Carmel A.
    Miller, Jeremy K.
    Moore, Katherine S.
    Meixner, Johannes M.
    Munafo, Marcus R.
    Neijenhuijs, Koen I.
    Nilsonne, Gustav
    Nosek, Brian A.
    Plessow, Franziska
    Prenoveau, Jason M.
    Ricker, Ashley A.
    Schmidt, Kathleen
    Spies, Jeffrey R.
    Stieger, Stefan
    Strohminger, Nina
    Sullivan, Gavin B.
    van Aert, Robbie C. M.
    van Assen, Marcel A. L. M.
    Vanpaemel, Wolf
    Vianello, Michelangelo
    Voracek, Martin
    Zuni, Kellylynn
    [J]. SCIENCE, 2016, 351 (6277)
  • [5] Normative dissonance in science: Results from a national survey of US scientists
    Anderson, Melissa S.
    Martinson, Brian C.
    De Vries, Raymond
    [J]. JOURNAL OF EMPIRICAL RESEARCH ON HUMAN RESEARCH ETHICS, 2007, 2 (04) : 3 - 14
  • [6] [Anonymous], 2018, Science
  • [7] Journal Article Reporting Standards for Quantitative Research in Psychology: The APA Publications and Communications Board Task Force Report
    Appelbaum, Mark
    Cooper, Harris
    Kline, Rex B.
    Mayo-Wilson, Evan
    Nezu, Arthur M.
    Rao, Stephen M.
    [J]. AMERICAN PSYCHOLOGIST, 2018, 73 (01) : 3 - 25
  • [8] Armeni K, 2020, METAARXIV, DOI [10.31222/osf.io/7gct9, DOI 10.31222/OSF.IO/7GCT9]
  • [9] The Reproducibility of Statistical Results in Psychological Research: An Investigation Using Unpublished Raw Data
    Artner, Richard
    Verliefde, Thomas
    Steegen, Sara
    Gomes, Sara
    Traets, Frits
    Tuerlinckx, Francis
    Vanpaemel, Wolf
    [J]. PSYCHOLOGICAL METHODS, 2021, 26 (05) : 527 - 546
  • [10] Baker, 2016, Nature, DOI [10.1038/nature.2016.20287, DOI 10.1038/NATURE.2016.20287]