Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies

被引：27

作者：

Aczel, Balazs ^{[1
]}

Szaszi, Barnabas ^{[1
]}

Nilsonne, Gustav ^{[2
,3
]}

van den Akker, Olmo R. ^{[4
]}

Albers, Casper J. ^{[5
]}

van Assen, Marcel Alm ^{[4
,6
]}

Bastiaansen, Jojanneke A. ^{[7
,8
]}

Benjamin, Daniel ^{[9
,10
]}

Boehm, Udo ^{[11
]}

Botvinik-Nezer, Rotem ^{[12
]}

Bringmann, Laura F. ^{[5
]}

Busch, Niko A. ^{[13
]}

Caruyer, Emmanuel ^{[14
,15
]}

Cataldo, Andrea M. ^{[16
,17
]}

Cowan, Nelson ^{[18
]}

Delios, Andrew ^{[19
]}

van Dongen, Noah N. N. ^{[11
]}

Donkin, Chris ^{[20
]}

van Doorn, Johnny B. ^{[11
]}

Dreber, Anna ^{[21
,22
]}

Dutilh, Gilles ^{[23
]}

Egan, Gary F. ^{[24
]}

Gernsbacher, Morton Ann ^{[25
]}

Hoekstra, Rink ^{[5
]}

Hoffmann, Sabine ^{[26
]}

Holzmeister, Felix ^{[22
]}

Huber, Juergen ^{[22
]}

Johannesson, Magnus ^{[21
]}

Jonas, Kai J. ^{[27
]}

Kindel, Alexander T. ^{[28
]}

Kirchler, Michael ^{[22
]}

Kunkels, Yoram K. ^{[7
]}

Lindsay, D. Stephen ^{[29
]}

Mangin, Jean-Francois ^{[30
,31
]}

Matzke, Dora ^{[32
]}

Munafo, Marcus R. ^{[33
]}

Newell, Ben R. ^{[20
]}

Nosek, Brian A. ^{[34
,35
]}

Poldrack, Russell A. ^{[36
]}

van Ravenzwaaij, Don ^{[5
]}

Rieskamp, Jorg ^{[37
]}

Salganik, Matthew J. ^{[28
]}

Sarafoglou, Alexandra ^{[32
]}

Schonberg, Tom ^{[38
]}

Schweinsberg, Martin ^{[39
]}

Shanks, David ^{[40
]}

Silberzahn, Raphael ^{[41
]}

Simons, Daniel J. ^{[42
]}

Spellman, Barbara A. ^{[35
]}

St-Jean, Samuel ^{[43
,44
]}

机构：

[1] Eotvos Lorand Univ, Budapest, Hungary

[2] Karolinska Inst, Stockholm, Sweden

[3] Stockholm Univ, Stockholm, Sweden

[4] Tilburg Univ, Tilburg, Netherlands

[5] Univ Groningen, Groningen, Netherlands

[6] Univ Utrecht, Utrecht, Netherlands

[7] Univ Groningen, Univ Med Ctr Groningen, Groningen, Netherlands

[8] Friesland Mental Hlth Care Serv, Leeuwarden, Netherlands

[9] Univ Calif Los Angeles, Los Angeles, CA USA

[10] NBER, Cambridge, MA 02138 USA

[11] Univ Amsterdam, Amsterdam, Netherlands

[12] Dartmouth Coll, Hanover, NH 03755 USA

[13] Univ Munster, Munster, Germany

[14] Univ Rennes, CNRS, INRIA, Rennes, France

[15] INSERM, Rennes, France

[16] McLean Hosp, 115 Mill St, Belmont, MA 02178 USA

[17] Harvard Med Sch, Boston, MA 02115 USA

[18] Univ Missouri, Columbia, MO USA

[19] Natl Univ Singapore, Singapore, Singapore

[20] Univ New South Wales, Sydney, NSW, Australia

[21] Stockholm Sch Econ, Stockholm, Sweden

[22] Univ Innsbruck, Innsbruck, Austria

[23] Univ Hosp Basel, Basel, Switzerland

[24] Monash Univ, Melbourne, Vic, Australia

[25] Univ Wisconsin, Madison, WI USA

[26] Ludwig Maximilians Univ Munchen, Munich, Germany

[27] Maastricht Univ, Maastricht, Netherlands

[28] Princeton Univ, Princeton, NJ 08544 USA

[29] Univ Victoria, Victoria, BC, Canada

[30] Univ Paris Saclay, Paris, France

[31] CEA, Neurospin, Paris, France

[32] Univ Amsterdam, Amsterdam, Netherlands

[33] Univ Bristol, Bristol, Avon, England

[34] Ctr Open Sci, Charlottesville, VA USA

[35] Univ Virginia, Charlottesville, VA USA

[36] Stanford Univ, Stanford, CA 94305 USA

[37] Univ Basel, Basel, Switzerland

[38] Tel Aviv Univ, Tel Aviv, Israel

[39] ESMT Berlin, Berlin, Germany

[40] UCL, London, England

[41] Univ Sussex, Brighton, E Sussex, England

[42] Univ Illinois, Urbana, IL USA

[43] Univ Alberta, Edmonton, AB, Canada

[44] Lund Univ, Lund, Sweden

[45] Univ Massachusetts, Amherst, MA 01003 USA

[46] INSEAD, Singapore, Singapore

来源：

ELIFE | 2021年 / 10卷

基金：

欧洲研究理事会; 美国国家科学基金会; 加拿大自然科学与工程研究理事会;

关键词：

multi-analyst; metascience; statistical practice; science forum; expert consensus; analytical variability; None; REPRODUCIBILITY; DELPHI;

D O I：

10.7554/eLife.72185

中图分类号：

Q [生物科学];

学科分类号：

07 ; 0710 ; 09 ;

摘要：

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

引用

页数：13

共 44 条

[1] Aczel B, 2020, NAT HUM BEHAV, V4, P4, DOI 10.1038/s41562-019-0772-6
[2] Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology
Bastiaansen, Jojanneke A.
Kunkels, Yoram K.
Blaauw, Frank J.
Boker, Steven M.
Ceulemans, Eva
Cheng, Meng
Chow, Sy-Miin
de Jonge, Peter
Emerencia, Ando C.
Epskamp, Sacha
Fisher, Aaron J.
Hamaker, Ellen L.
Kuppens, Peter
Lutz, Wolfgang
Meyer, M. Joseph
Moulder, Robert
Oravecz, Zita
Riese, Harriette
Rubel, Julian
Ryan, Oisin
Servaas, Michelle N.
Sjobeck, Gustav
Snippe, Evelien
Trull, Timothy J.
Tschacher, Wolfgang
van der Veen, Date C.
Wichers, Marieke
Wood, Phillip K.
Woods, William C.
Wright, Aidan G. C.
Albers, Casper J.
Bringmann, Laura F.
[J]. JOURNAL OF PSYCHOSOMATIC RESEARCH, 2020, 137
[3] Boettiger Carl, 2015, ACM SIGOPS Operating Systems Review, V49, P71
[4] Variability in the analysis of a single neuroimaging dataset by many teams
Botvinik-Nezer, Rotem
Holzmeister, Felix
Camerer, Colin F.
Dreber, Anna
Huber, Juergen
Johannesson, Magnus
Kirchler, Michael
Iwanir, Roni
Mumford, Jeanette A.
Adcock, R. Alison
Avesani, Paolo
Baczkowski, Blazej M.
Bajracharya, Aahana
Bakst, Leah
Ball, Sheryl
Barilari, Marco
Bault, Nadege
Beaton, Derek
Beitner, Julia
Benoit, Roland G.
Berkers, Ruud M. W. J.
Bhanji, Jamil P.
Biswal, Bharat B.
Bobadilla-Suarez, Sebastian
Bortolini, Tiago
Bottenhorn, Katherine L.
Bowring, Alexander
Braem, Senne
Brooks, Hayley R.
Brudner, Emily G.
Calderon, Cristian B.
Camilleri, Julia A.
Castrellon, Jaime J.
Cecchetti, Luca
Cieslik, Edna C.
Cole, Zachary J.
Collignon, Olivier
Cox, Robert W.
Cunningham, William A.
Czoschke, Stefan
Dadi, Kamalaker
Davis, Charles P.
Luca, Alberto De
Delgado, Mauricio R.
Demetriou, Lysia
Dennison, Jeffrey B.
Di, Xin
Dickie, Erin W.
Dobryakova, Ekaterina
Donnat, Claire L.
[J]. NATURE, 2020, 582 (7810) : 84 - +
[5] Registered Reports: A new publishing initiative at Cortex
Chambers, Christopher D.
[J]. CORTEX, 2013, 49 (03) : 609 - 610
[6] Same Data, Opposite Results? A Call to Improve Surgical Database Research
Childers, Christopher P.
Maggard-Gibbons, Melinda
[J]. JAMA SURGERY, 2021, 156 (03) : 219 - 220
[7] The Unconscious Thought Effect in Clinical Decision Making: An Example in Diagnosis
de Vries, Marieke
Witteman, Cilia L. M.
Holland, Rob W.
Dijksterhuis, Ap
[J]. MEDICAL DECISION MAKING, 2010, 30 (05) : 578 - 581
[8] Increasing the Transparency of Research Papers with Explorable Multiverse Analyses
Dragicevic, Pierre
Jansen, Yvonne
Sarma, Abhraneel
Kay, Matthew
Chevalier, Fanny
[J]. CHI 2019: PROCEEDINGS OF THE 2019 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2019,
[9] Drechsler J., 2011, Synthetic Datasets for Statistical Disclosure Control, V53, DOI DOI 10.1007/978-1-4614-0326-5
[10] The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models
Dutilh, Gilles
Annis, Jeffrey
Brown, Scott D.
Cassey, Peter
Evans, Nathan J.
Grasman, Raoul P. P. P.
Hawkins, Guy E.
Heathcote, Andrew
Holmes, William R.
Krypotos, Angelos-Miltiadis
Kupitz, Colin N.
Leite, Fabio P.
Lerche, Veronika
Lin, Yi-Shin
Logan, Gordon D.
Palmeri, Thomas J.
Starns, Jeffrey J.
Trueblood, Jennifer S.
van Maanen, Leendert
van Ravenzwaaij, Don
Vandekerckhove, Joachim
Visser, Ingmar
Voss, Andreas
White, Corey N.
Wiecki, Thomas V.
Rieskamp, Joerg
Donkin, Chris
[J]. PSYCHONOMIC BULLETIN & REVIEW, 2019, 26 (04) : 1051 - 1069

← 1 2 3 4 5 →