Threats of a Replication Crisis in Empirical Computer Science

被引：80

作者：

Cockburn, Andy ^{[1
]}

Dragicevic, Pierre ^{[2
]}

Besancon, Lonni ^{[3
]}

Gutwin, Carl ^{[4
,5
]}

机构：

[1] Univ Canterbury, HCI & Multimedia Lab, Christchurch, New Zealand

[2] Inria, Orsay, France

[3] Linkoping Univ, Linkoping, Sweden

[4] Univ Saskatchewan, Dept Comp Sci, Saskatoon, SK, Canada

[5] Univ Saskatchewan, HCI Lab, Saskatoon, SK, Canada

来源：

COMMUNICATIONS OF THE ACM | 2020年 / 63卷 / 08期

关键词：

REPEATABILITY; OUTCOMES;

D O I：

10.1145/3360311

中图分类号：

TP3 [计算技术、计算机技术];

学科分类号：

0812 ;

摘要：

引用

页码：70 / 79

页数：10

共 42 条

[1] Estimating the reproducibility of psychological science [J].

Aarts, Alexander A. ;

Anderson, Joanna E. ;

Anderson, Christopher J. ;

Attridge, Peter R. ;

Attwood, Angela ;

Axt, Jordan ;

Babel, Molly ;

Bahnik, Stepan ;

Baranski, Erica ;

Barnett-Cowan, Michael ;

Bartmess, Elizabeth ;

Beer, Jennifer ;

Bell, Raoul ;

Bentley, Heather ;

Beyan, Leah ;

Binion, Grace ;

Borsboom, Denny ;

Bosch, Annick ;

Bosco, Frank A. ;

Bowman, Sara D. ;

Brandt, Mark J. ;

Braswell, Erin ;

Brohmer, Hilmar ;

Brown, Benjamin T. ;

Brown, Kristina ;

Bruening, Jovita ;

Calhoun-Sauls, Ann ;

Callahan, Shannon P. ;

Chagnon, Elizabeth ;

Chandler, Jesse ;

Chartier, Christopher R. ;

Cheung, Felix ;

Christopherson, Cody D. ;

Cillessen, Linda ;

Clay, Russ ;

Cleary, Hayley ;

Cloud, Mark D. ;

Cohn, Michael ;

Cohoon, Johanna ;

Columbus, Simon ;

Cordes, Andreas ;

Costantini, Giulio ;

Alvarez, Leslie D. Cramblet ;

Cremata, Ed ;

Crusius, Jan ;

DeCoster, Jamie ;

DeGaetano, Michelle A. ;

Della Penna, Nicolas ;

den Bezemer, Bobby ;

Deserno, Marie K. .

SCIENCE, 2015, 349 (6251)

[2] The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research [J].

Amrhein, Valentin ;

Korner-Nievergelt, Franzi ;

Roth, Tobias .

PEERJ, 2017, 5

[3]

[Anonymous], 2012, UNDERSTANDING NEW ST

[4]

Baker M, 2016, NATURE, V533, P452, DOI 10.1038/533452a

[5] Raise standards for preclinical cancer research [J].

Begley, C. Glenn ;

Ellis, Lee M. .

NATURE, 2012, 483 (7391) :531-533

[6] Redefine statistical significance [J].

Benjamin, Daniel J. ;

Berger, James O. ;

Johannesson, Magnus ;

Nosek, Brian A. ;

Wagenmakers, E. -J. ;

Berk, Richard ;

Bollen, Kenneth A. ;

Brembs, Bjoern ;

Brown, Lawrence ;

Camerer, Colin ;

Cesarini, David ;

Chambers, Christopher D. ;

Clyde, Merlise ;

Cook, Thomas D. ;

De Boeck, Paul ;

Dienes, Zoltan ;

Dreber, Anna ;

Easwaran, Kenny ;

Efferson, Charles ;

Fehr, Ernst ;

Fidler, Fiona ;

Field, Andy P. ;

Forster, Malcolm ;

George, Edward I. ;

Gonzalez, Richard ;

Goodman, Steven ;

Green, Edwin ;

Green, Donald P. ;

Greenwald, Anthony ;

Hadfield, Jarrod D. ;

Hedges, Larry V. ;

Held, Leonhard ;

Ho, Teck Hua ;

Hoijtink, Herbert ;

Hruschka, Daniel J. ;

Imai, Kosuke ;

Imbens, Guido ;

Ioannidis, John P. A. ;

Jeon, Minjeong ;

Jones, James Holland ;

Kirchler, Michael ;

Laibson, David ;

List, John ;

Little, Roderick ;

Lupia, Arthur ;

Machery, Edouard ;

Maxwell, Scott E. ;

McCarthy, Michael ;

Moore, Don ;

Morgan, Stephen L. .

NATURE HUMAN BEHAVIOUR, 2018, 2 (01) :6-10

[7] Incentivizing Reproducibility [J].

Boisvert, Ronald F. .

COMMUNICATIONS OF THE ACM, 2016, 59 (10) :5-5

[8] CRITICAL QUESTIONS FOR BIG DATA Provocations for a cultural, technological, and scholarly phenomenon [J].

Boyd, Danah ;

Crawford, Kate .

INFORMATION COMMUNICATION & SOCIETY, 2012, 15 (05) :662-679

[9] Koopman Invariant Subspaces and Finite Linear Representations of Nonlinear Dynamical Systems for Control [J].

Brunton, Steven L. ;

Brunton, Bingni W. ;

Proctor, Joshua L. ;

Kutz, J. Nathan .

PLOS ONE, 2016, 11 (02)

[10]

Cockburn A., 2018, P 2018 CHI C HUM FAC

← 1 2 3 4 5 →