Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

被引：89

作者：

Breznau, Nate ^{[1
]}

Rinke, Eike Mark ^{[2
]}

Wuttke, Alexander ^{[3
]}

Nguyen, Hung H. V. ^{[1
,4
]}

Adem, Muna ^{[5
]}

Adriaans, Jule ^{[6
]}

Alvarez-Benjumea, Amalia ^{[7
]}

Andersen, Henrik K. ^{[8
]}

Auer, Daniel ^{[3
]}

Azevedo, Flavio ^{[10
]}

Bahnsen, Oke ^{[9
]}

Balzer, Dave ^{[11
]}

Bauer, Gerrit ^{[93
]}

Bauer, Paul C. ^{[3
]}

Baumann, Markus ^{[13
,29
]}

Baute, Sharon ^{[14
]}

Benoit, Verena ^{[12
,15
]}

Bernauer, Julian ^{[3
]}

Berning, Carl ^{[94
]}

Berthold, Anna ^{[15
]}

Bethke, Felix S. ^{[16
]}

Biegert, Thomas ^{[17
]}

Blinzler, Katharina ^{[18
]}

Blumenberg, Johannes N. ^{[96
]}

Bobzien, Licia ^{[19
]}

Bohman, Andrea ^{[20
]}

Bol, Thijs ^{[24
,99
]}

Bostic, Amie ^{[21
]}

Brzozowska, Zuzanna ^{[22
,23
]}

Burgdorf, Katharina ^{[9
]}

Burger, Kaspar ^{[24
,25
,100
]}

Busch, Kathrin B.

Carlos-Castillo, Juan ^{[26
,91
]}

Chan, Nathan ^{[92
]}

Christmann, Pablo ^{[97
]}

Connelly, Roxanne ^{[28
]}

Czymara, Christian S. ^{[95
]}

Damian, Elena ^{[103
]}

Ecker, Alejandro ^{[3
]}

Edelmann, Achim ^{[31
]}

Eger, Maureen A. ^{[20
]}

Ellerbrock, Simon ^{[3
,9
]}

Forke, Anna

Forster, Andrea ^{[41
]}

Gaasendam, Chris ^{[30
]}

Gavras, Konstantin ^{[9
]}

Gayle, Vernon ^{[28
]}

Gessler, Theresa ^{[86
]}

Gnambs, Timo ^{[32
]}

Godefroidt, Amelie ^{[105
]}

机构：

[1] Univ Bremen, Res Ctr Inequal & Social Policy SOCIUM, D-28359 Bremen, Germany

[2] Univ Leeds, Sch Polit & Int Studies, Leeds LS2 9JT, W Yorkshire, England

[3] Univ Mannheim, Mannheim Ctr European Social Res, D-68131 Mannheim, Germany

[4] Bremen Int Grad Sch Social Sci, D-28359 Bremen, Germany

[5] Indiana Univ, Dept Sociol, Bloomington, IN 47405 USA

[6] German Inst Econ Res DIW, Socioecon Panel Study SOEP, D-10117 Berlin, Germany

[7] Max Planck Inst Res Collect Goods, Mech Normat Change, D-53113 Bonn, Germany

[8] Tech Univ Chemnitz, Inst Sociol, D-09126 Chemnitz, Germany

[9] Univ Mannheim, Sch Social Sci, D-68159 Mannheim, Germany

[10] Univ Cambridge, Dept Psychol, Cambridge CB2 3RQ, England

[11] Johannes Gutenberg Univ Mainz, Inst Sociol, D-55128 Mainz, Germany

[12] Ludwig Maximilians Univ Munchen, Dept Polit Sci, D-80539 Munich, Germany

[13] Heidelberg Univ, D-69117 Heidelberg, Germany

[14] Univ Konstanz, Comparat Polit Econ, D-78457 Constance, Germany

[15] Univ Bamberg, Fac Social Sci Econ & Business Adm, D-96052 Bamberg, Germany

[16] Peace Res Inst Frankfurt, Res Dept Intrastate Conflict, D-60329 Frankfurt, Germany

[17] London Sch Econ & Polit Sci, Dept Social Policy, London WC2A 2AE, England

[18] Leibniz Inst Social Sci GESIS, Survey Data Curat, D-50667 Cologne, Germany

[19] Hertie Sch, Jacques Delors Ctr, D-10117 Berlin, Germany

[20] Umea Univ, Dept Sociol, S-90187 Umea, Sweden

[21] Univ Texas Rio Grande Valley, Dept Sociol, Brownsville, TX 78520 USA

[22] Austrian Acad Sci, Vienna Inst Demog, A-1030 Vienna, Austria

[23] Gesundheit Osterreich GOG, Austrian Natl Publ Hlth Inst, A-1030 Vienna, Austria

[24] UCL, Social Res Inst, Inst Educ, London WC1H 0AL, England

[25] Univ Zurich, Dept Sociol, CH-8050 Zurich, Switzerland

[26] Univ Chile, Dept Sociol, Santiago 7800284, Chile

[27] Univ Calif Irvine, Dept Polit Sci, Irvine, CA 92617 USA

[28] Univ Edinburgh, Sch Social & Polit Sci, Edinburgh EH8 9LD, Midlothian, Scotland

[29] Goethe Univ Frankfurt, Inst Polit Sci, D-60323 Frankfurt, Germany

[30] Katholieke Univ Leuven, Ctr Sociol Res, Dept Sociol, B-3000 Leuven, Belgium

[31] Medialab Sci Po, F-75007 Paris, France

[32] Leibniz Inst Educ Trajectories, Educ Measurement, D-96047 Bamberg, Germany

[33] Griffith Univ, Sch Govt & Int Relat, Nathan, Qld 4111, Australia

[34] Univ Tubingen, Dept Sociol, D-72074 Tubingen, Germany

[35] Max Planck Inst Social Law & Social Policy, D-80799 Munich, Germany

[36] Univ Luxembourg, L-4365 Esch Sur Alzette, Luxembourg

[37] Univ Libre Bruxelles, Dept Polit Sci, B-1050 Brussels, Belgium

[38] Hans Bockler Fdn, Wirtschafts & Sozialwissensch Inst WSI, D-40474 Dusseldorf, Germany

[39] Humboldt Univ, Berlin Inst Integrat & Migrat Res BIM, D-10099 Berlin, Germany

[40] Univ Wuppertal, Sch Human & Social Sci, D-42119 Wuppertal, Germany

[41] Free Univ Berlin, Empir Educ & Higher Educ Res, D-14195 Berlin, Germany

[42] German Socio Econ Panel Survey, D-10117 Berlin, Germany

[43] Tilburg Univ, Dept Social Psychol, NL-5037 AB Tilburg, Netherlands

[44] Univ Duisburg Essen, Inst Socioecon, D-47057 Duisburg, Germany

[45] Zeppelin Univ, D-88045 Friedrichshafen, Germany

[46] Med Sch Hamburg, Dept Psychol, D-20457 Hamburg, Germany

[47] Fed Stat Off Germany, D-65189 Wiesbaden, Germany

[48] Polish Acad Sci, Inst Polit Studies, Dept Res Social & Inst Transformat, PL-00625 Warsaw, Poland

[49] Univ Oklahoma, Dept Polit Sci, Norman, OK 73019 USA

[50] Univ Oslo, Dept Polit Sci, N-0851 Oslo, Norway

来源：

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA | 2022年 / 119卷 / 44期

关键词：

metascience; many analysts; researcher degrees of freedom; analytical flexibility; immigration and policy preferences; WELFARE-STATE; IMMIGRATION; SUPPORT; REDISTRIBUTION; PREFERENCES; ANALYSTS; IDEAS;

D O I：

10.1073/pnas.2203150119

中图分类号：

O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];

学科分类号：

07 ; 0710 ; 09 ;

摘要：

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

引用

页数：8

共 34 条

[11] Breznau Nate, 2021, HarvardDataverse, V1, DOI 10.7910/DVN/UUP8CX
[12] Multilevel Modelling of Country Effects: A Cautionary Tale
Bryan, Mark L.
Jenkins, Stephen P.
[J]. EUROPEAN SOCIOLOGICAL REVIEW, 2016, 32 (01) : 3 - 22
[13] IMMIGRATION, INTEGRATION, AND SUPPORT FOR REDISTRIBUTION IN EUROPE
Burgoon, Brian
[J]. WORLD POLITICS, 2014, 66 (03) : 365 - +
[14] Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015
Camerer, Colin F.
Dreber, Anna
Holzmeister, Felix
Ho, Teck-Hua
Huber, Juegen
Johannesson, Magnus
Kirchler, Michael
Nave, Gideon
Nosek, Brian A.
Pfeiffer, Thomas
Altmejd, Adam
Buttrick, Nick
Chan, Taizan
Chen, Yiling
Forsell, Eskil
Gampa, Anup
Heikensten, Emma
Hummer, Lily
Imai, Taisuke
Isaksson, Siri
Manfredi, Dylan
Rose, Julia
Wagenmakers, Eric-Jan
Wu, Hang
[J]. NATURE HUMAN BEHAVIOUR, 2018, 2 (09): : 637 - 644
[15] Del Giudice M., 2021, ADV METH PRACT PSYCH, DOI [10.1177/2515245920954925, DOI 10.1177/2515245920954925]
[16] The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models
Dutilh, Gilles
Annis, Jeffrey
Brown, Scott D.
Cassey, Peter
Evans, Nathan J.
Grasman, Raoul P. P. P.
Hawkins, Guy E.
Heathcote, Andrew
Holmes, William R.
Krypotos, Angelos-Miltiadis
Kupitz, Colin N.
Leite, Fabio P.
Lerche, Veronika
Lin, Yi-Shin
Logan, Gordon D.
Palmeri, Thomas J.
Starns, Jeffrey J.
Trueblood, Jennifer S.
van Maanen, Leendert
van Ravenzwaaij, Don
Vandekerckhove, Joachim
Visser, Ingmar
Voss, Andreas
White, Corey N.
Wiecki, Thomas V.
Rieskamp, Joerg
Donkin, Chris
[J]. PSYCHONOMIC BULLETIN & REVIEW, 2019, 26 (04) : 1051 - 1069
[17] Immigration and the welfare state: A cross-regional analysis of European welfare attitudes
Eger, Maureen A.
Breznau, Nate
[J]. INTERNATIONAL JOURNAL OF COMPARATIVE SOCIOLOGY, 2017, 58 (05) : 440 - 463
[18] Even in Sweden: The Effect of Immigration on Support for Welfare State Spending
Eger, Maureen A.
[J]. EUROPEAN SOCIOLOGICAL REVIEW, 2010, 26 (02) : 203 - 217
[19] The Emergence of Statistical Objectivity: Changing Ideas of Epistemic Vice and Virtue in Science
Freese, Jeremy
Peterson, David
[J]. SOCIOLOGICAL THEORY, 2018, 36 (03) : 289 - 313
[20] Publishing as prostitution? - Choosing between one's own ideas and academic success
Frey, BS
[J]. PUBLIC CHOICE, 2003, 116 (1-2) : 205 - 223

← 1 2 3 4 →