Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

被引：89

作者：

Breznau, Nate ^{[1
]}

Rinke, Eike Mark ^{[2
]}

Wuttke, Alexander ^{[3
]}

Nguyen, Hung H. V. ^{[1
,4
]}

Adem, Muna ^{[5
]}

Adriaans, Jule ^{[6
]}

Alvarez-Benjumea, Amalia ^{[7
]}

Andersen, Henrik K. ^{[8
]}

Auer, Daniel ^{[3
]}

Azevedo, Flavio ^{[10
]}

Bahnsen, Oke ^{[9
]}

Balzer, Dave ^{[11
]}

Bauer, Gerrit ^{[93
]}

Bauer, Paul C. ^{[3
]}

Baumann, Markus ^{[13
,29
]}

Baute, Sharon ^{[14
]}

Benoit, Verena ^{[12
,15
]}

Bernauer, Julian ^{[3
]}

Berning, Carl ^{[94
]}

Berthold, Anna ^{[15
]}

Bethke, Felix S. ^{[16
]}

Biegert, Thomas ^{[17
]}

Blinzler, Katharina ^{[18
]}

Blumenberg, Johannes N. ^{[96
]}

Bobzien, Licia ^{[19
]}

Bohman, Andrea ^{[20
]}

Bol, Thijs ^{[24
,99
]}

Bostic, Amie ^{[21
]}

Brzozowska, Zuzanna ^{[22
,23
]}

Burgdorf, Katharina ^{[9
]}

Burger, Kaspar ^{[24
,25
,100
]}

Busch, Kathrin B.

Carlos-Castillo, Juan ^{[26
,91
]}

Chan, Nathan ^{[92
]}

Christmann, Pablo ^{[97
]}

Connelly, Roxanne ^{[28
]}

Czymara, Christian S. ^{[95
]}

Damian, Elena ^{[103
]}

Ecker, Alejandro ^{[3
]}

Edelmann, Achim ^{[31
]}

Eger, Maureen A. ^{[20
]}

Ellerbrock, Simon ^{[3
,9
]}

Forke, Anna

Forster, Andrea ^{[41
]}

Gaasendam, Chris ^{[30
]}

Gavras, Konstantin ^{[9
]}

Gayle, Vernon ^{[28
]}

Gessler, Theresa ^{[86
]}

Gnambs, Timo ^{[32
]}

Godefroidt, Amelie ^{[105
]}

机构：

[1] Univ Bremen, Res Ctr Inequal & Social Policy SOCIUM, D-28359 Bremen, Germany

[2] Univ Leeds, Sch Polit & Int Studies, Leeds LS2 9JT, W Yorkshire, England

[3] Univ Mannheim, Mannheim Ctr European Social Res, D-68131 Mannheim, Germany

[4] Bremen Int Grad Sch Social Sci, D-28359 Bremen, Germany

[5] Indiana Univ, Dept Sociol, Bloomington, IN 47405 USA

[6] German Inst Econ Res DIW, Socioecon Panel Study SOEP, D-10117 Berlin, Germany

[7] Max Planck Inst Res Collect Goods, Mech Normat Change, D-53113 Bonn, Germany

[8] Tech Univ Chemnitz, Inst Sociol, D-09126 Chemnitz, Germany

[9] Univ Mannheim, Sch Social Sci, D-68159 Mannheim, Germany

[10] Univ Cambridge, Dept Psychol, Cambridge CB2 3RQ, England

[11] Johannes Gutenberg Univ Mainz, Inst Sociol, D-55128 Mainz, Germany

[12] Ludwig Maximilians Univ Munchen, Dept Polit Sci, D-80539 Munich, Germany

[13] Heidelberg Univ, D-69117 Heidelberg, Germany

[14] Univ Konstanz, Comparat Polit Econ, D-78457 Constance, Germany

[15] Univ Bamberg, Fac Social Sci Econ & Business Adm, D-96052 Bamberg, Germany

[16] Peace Res Inst Frankfurt, Res Dept Intrastate Conflict, D-60329 Frankfurt, Germany

[17] London Sch Econ & Polit Sci, Dept Social Policy, London WC2A 2AE, England

[18] Leibniz Inst Social Sci GESIS, Survey Data Curat, D-50667 Cologne, Germany

[19] Hertie Sch, Jacques Delors Ctr, D-10117 Berlin, Germany

[20] Umea Univ, Dept Sociol, S-90187 Umea, Sweden

[21] Univ Texas Rio Grande Valley, Dept Sociol, Brownsville, TX 78520 USA

[22] Austrian Acad Sci, Vienna Inst Demog, A-1030 Vienna, Austria

[23] Gesundheit Osterreich GOG, Austrian Natl Publ Hlth Inst, A-1030 Vienna, Austria

[24] UCL, Social Res Inst, Inst Educ, London WC1H 0AL, England

[25] Univ Zurich, Dept Sociol, CH-8050 Zurich, Switzerland

[26] Univ Chile, Dept Sociol, Santiago 7800284, Chile

[27] Univ Calif Irvine, Dept Polit Sci, Irvine, CA 92617 USA

[28] Univ Edinburgh, Sch Social & Polit Sci, Edinburgh EH8 9LD, Midlothian, Scotland

[29] Goethe Univ Frankfurt, Inst Polit Sci, D-60323 Frankfurt, Germany

[30] Katholieke Univ Leuven, Ctr Sociol Res, Dept Sociol, B-3000 Leuven, Belgium

[31] Medialab Sci Po, F-75007 Paris, France

[32] Leibniz Inst Educ Trajectories, Educ Measurement, D-96047 Bamberg, Germany

[33] Griffith Univ, Sch Govt & Int Relat, Nathan, Qld 4111, Australia

[34] Univ Tubingen, Dept Sociol, D-72074 Tubingen, Germany

[35] Max Planck Inst Social Law & Social Policy, D-80799 Munich, Germany

[36] Univ Luxembourg, L-4365 Esch Sur Alzette, Luxembourg

[37] Univ Libre Bruxelles, Dept Polit Sci, B-1050 Brussels, Belgium

[38] Hans Bockler Fdn, Wirtschafts & Sozialwissensch Inst WSI, D-40474 Dusseldorf, Germany

[39] Humboldt Univ, Berlin Inst Integrat & Migrat Res BIM, D-10099 Berlin, Germany

[40] Univ Wuppertal, Sch Human & Social Sci, D-42119 Wuppertal, Germany

[41] Free Univ Berlin, Empir Educ & Higher Educ Res, D-14195 Berlin, Germany

[42] German Socio Econ Panel Survey, D-10117 Berlin, Germany

[43] Tilburg Univ, Dept Social Psychol, NL-5037 AB Tilburg, Netherlands

[44] Univ Duisburg Essen, Inst Socioecon, D-47057 Duisburg, Germany

[45] Zeppelin Univ, D-88045 Friedrichshafen, Germany

[46] Med Sch Hamburg, Dept Psychol, D-20457 Hamburg, Germany

[47] Fed Stat Off Germany, D-65189 Wiesbaden, Germany

[48] Polish Acad Sci, Inst Polit Studies, Dept Res Social & Inst Transformat, PL-00625 Warsaw, Poland

[49] Univ Oklahoma, Dept Polit Sci, Norman, OK 73019 USA

[50] Univ Oslo, Dept Polit Sci, N-0851 Oslo, Norway

来源：

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA | 2022年 / 119卷 / 44期

关键词：

metascience; many analysts; researcher degrees of freedom; analytical flexibility; immigration and policy preferences; WELFARE-STATE; IMMIGRATION; SUPPORT; REDISTRIBUTION; PREFERENCES; ANALYSTS; IDEAS;

D O I：

10.1073/pnas.2203150119

中图分类号：

O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];

学科分类号：

07 ; 0710 ; 09 ;

摘要：

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

引用

页数：8

共 34 条

[1] Estimating the reproducibility of psychological science
Aarts, Alexander A.
Anderson, Joanna E.
Anderson, Christopher J.
Attridge, Peter R.
Attwood, Angela
Axt, Jordan
Babel, Molly
Bahnik, Stepan
Baranski, Erica
Barnett-Cowan, Michael
Bartmess, Elizabeth
Beer, Jennifer
Bell, Raoul
Bentley, Heather
Beyan, Leah
Binion, Grace
Borsboom, Denny
Bosch, Annick
Bosco, Frank A.
Bowman, Sara D.
Brandt, Mark J.
Braswell, Erin
Brohmer, Hilmar
Brown, Benjamin T.
Brown, Kristina
Bruening, Jovita
Calhoun-Sauls, Ann
Callahan, Shannon P.
Chagnon, Elizabeth
Chandler, Jesse
Chartier, Christopher R.
Cheung, Felix
Christopherson, Cody D.
Cillessen, Linda
Clay, Russ
Cleary, Hayley
Cloud, Mark D.
Cohn, Michael
Cohoon, Johanna
Columbus, Simon
Cordes, Andreas
Costantini, Giulio
Alvarez, Leslie D. Cramblet
Cremata, Ed
Crusius, Jan
DeCoster, Jamie
DeGaetano, Michelle A.
Della Penna, Nicolas
den Bezemer, Bobby
Deserno, Marie K.
[J]. SCIENCE, 2015, 349 (6251)
[2] Alesina A., 2004, FIGHTING POVERTY US
[3] Immigration and preferences for redistribution in Europe
Alesina, Alberto
Murard, Elie
Rapoport, Hillel
[J]. JOURNAL OF ECONOMIC GEOGRAPHY, 2021, 21 (06) : 925 - 954
[4] Alesina Alberto., 2006, Horizons strategiques, V2, P51
[5] Inequality, Labor Market Segmentation, and Preferences for Redistribution
Alt, James
Iversen, Torben
[J]. AMERICAN JOURNAL OF POLITICAL SCIENCE, 2017, 61 (01) : 21 - 36
[6] Auspurg K., 2021, SOCIUS, V86, P532
[7] Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology
Bastiaansen, Jojanneke A.
Kunkels, Yoram K.
Blaauw, Frank J.
Boker, Steven M.
Ceulemans, Eva
Cheng, Meng
Chow, Sy-Miin
de Jonge, Peter
Emerencia, Ando C.
Epskamp, Sacha
Fisher, Aaron J.
Hamaker, Ellen L.
Kuppens, Peter
Lutz, Wolfgang
Meyer, M. Joseph
Moulder, Robert
Oravecz, Zita
Riese, Harriette
Rubel, Julian
Ryan, Oisin
Servaas, Michelle N.
Sjobeck, Gustav
Snippe, Evelien
Trull, Timothy J.
Tschacher, Wolfgang
van der Veen, Date C.
Wichers, Marieke
Wood, Phillip K.
Woods, William C.
Wright, Aidan G. C.
Albers, Casper J.
Bringmann, Laura F.
[J]. JOURNAL OF PSYCHOSOMATIC RESEARCH, 2020, 137
[8] Variability in the analysis of a single neuroimaging dataset by many teams
Botvinik-Nezer, Rotem
Holzmeister, Felix
Camerer, Colin F.
Dreber, Anna
Huber, Juergen
Johannesson, Magnus
Kirchler, Michael
Iwanir, Roni
Mumford, Jeanette A.
Adcock, R. Alison
Avesani, Paolo
Baczkowski, Blazej M.
Bajracharya, Aahana
Bakst, Leah
Ball, Sheryl
Barilari, Marco
Bault, Nadege
Beaton, Derek
Beitner, Julia
Benoit, Roland G.
Berkers, Ruud M. W. J.
Bhanji, Jamil P.
Biswal, Bharat B.
Bobadilla-Suarez, Sebastian
Bortolini, Tiago
Bottenhorn, Katherine L.
Bowring, Alexander
Braem, Senne
Brooks, Hayley R.
Brudner, Emily G.
Calderon, Cristian B.
Camilleri, Julia A.
Castrellon, Jaime J.
Cecchetti, Luca
Cieslik, Edna C.
Cole, Zachary J.
Collignon, Olivier
Cox, Robert W.
Cunningham, William A.
Czoschke, Stefan
Dadi, Kamalaker
Davis, Charles P.
Luca, Alberto De
Delgado, Mauricio R.
Demetriou, Lysia
Dennison, Jeffrey B.
Di, Xin
Dickie, Erin W.
Dobryakova, Ekaterina
Donnat, Claire L.
[J]. NATURE, 2020, 582 (7810) : 84 - +
[9] Does Immigration Undermine Public Support for Social Policy?
Brady, David
Finnigan, Ryan
[J]. AMERICAN SOCIOLOGICAL REVIEW, 2014, 79 (01) : 17 - 42
[10] Breznau N., HIDDEN UNIVERSE DATA

← 1 2 3 4 →