Why rankings of biomedical image analysis competitions should be interpreted with care

被引：202

作者：

Maier-Hein, Lena ^{[1
]}

Eisenmann, Matthias ^{[1
]}

Reinke, Annika ^{[1
]}

Onogur, Sinan ^{[1
]}

Stankovic, Marko ^{[1
]}

Scholz, Patrick ^{[1
]}

Arbel, Tal ^{[2
]}

Bogunovic, Hrvoje ^{[3
]}

Bradley, Andrew P. ^{[4
]}

Carass, Aaron ^{[5
]}

Feldmann, Carolin ^{[1
]}

Frangi, Alejandro F. ^{[6
]}

Full, Peter M. ^{[1
]}

van Ginneken, Bram ^{[7
]}

Hanbury, Allan ^{[8
,9
]}

Honauer, Katrin ^{[10
]}

Kozubek, Michal ^{[11
]}

Landman, Bennett A. ^{[12
]}

Marz, Keno ^{[1
]}

Maier, Oskar ^{[13
]}

Maier-Hein, Klaus ^{[14
]}

Menze, Bjoern H. ^{[15
]}

Muller, Henning ^{[16
]}

Neher, Peter F. ^{[14
]}

Niessen, Wiro ^{[17
]}

Rajpoot, Nasir ^{[18
,19
]}

Sharp, Gregory C. ^{[20
]}

Sirinukunwattana, Korsuk ^{[21
]}

Speidel, Stefanie ^{[22
]}

Stock, Christian ^{[23
]}

Stoyanov, Danail ^{[24
]}

Taha, Abdel Aziz ^{[25
,26
]}

Van der Sommen, Fons ^{[27
]}

Wang, Ching-Wei ^{[28
]}

Weber, Marc-Andre ^{[29
]}

Zheng, Guoyan ^{[30
]}

Jannin, Pierre ^{[31
]}

Kopp-Schneider, Annette ^{[32
,33
]}

机构：

[1] German Canc Res Ctr, Div Comp Assisted Med Intervent CAMI, D-69120 Heidelberg, Germany

[2] McGill Univ, Ctr Intelligent Machines, Montreal, PQ H3A 0G4, Canada

[3] Med Univ Vienna, Dept Ophthalmol, Christian Doppler Lab Ophthalm Image Anal, A-1090 Vienna, Austria

[4] Queensland Univ Technol, Sci & Engn Fac, Brisbane, Qld 4001, Australia

[5] Johns Hopkins Univ, Dept Comp Sci, Dept Elect & Comp Engn, Baltimore, MD 21218 USA

[6] Univ Leeds, CISTIB Ctr Computat Imaging & Simulat Technol Bio, Leeds LS2 9JT, W Yorkshire, England

[7] Radboud Univ Nijmegen, Dept Radiol & Nucl Med, Med Image Anal, NL-6525 GA Nijmegen, Netherlands

[8] TU Wien, Inst Informat Syst Engn, A-1040 Vienna, Austria

[9] Complex Sci Hub Vienna, A-1080 Vienna, Austria

[10] Heidelberg Univ, Heidelberg Collaboratory Image Proc HCI, D-69120 Heidelberg, Germany

[11] Masaryk Univ, Ctr Biomed Image Anal, Brno 60200, Czech Republic

[12] Vanderbilt Univ, Elect Engn, Nashville, TN 37235 USA

[13] Univ Lubeck, Inst Med Informat, D-23562 Lubeck, Germany

[14] German Canc Res Ctr, Div Med Image Comp MIC, D-69120 Heidelberg, Germany

[15] Tech Univ Munich, Dept Informat, Inst Adv Studies, D-80333 Munich, Germany

[16] HES SO, Informat Syst Inst, CH-3960 Sierre, Switzerland

[17] Erasmus MC, Dept Radiol, NL-3015 GD Rotterdam, Netherlands

[18] Erasmus MC, Dept Nucl Med, NL-3015 GD Rotterdam, Netherlands

[19] Erasmus MC, Dept Med Informat, NL-3015 GD Rotterdam, Netherlands

[20] Univ Warwick, Dept Comp Sci, Coventry CV4 7AL, W Midlands, England

[21] Massachusetts Gen Hosp, Dept Radiat Oncol, Boston, MA 02114 USA

[22] Univ Oxford, Inst Biomed Engn, Oxford OX3 7DQ, England

[23] Natl Ctr Tumor Dis Dresden, Div Translat Surg Oncol TCO, D-01307 Dresden, Germany

[24] German Canc Res Ctr, Div Clin Epidemiol & Aging Res, D-69120 Heidelberg, Germany

[25] UCL, CMIC, London W1W 7TS, England

[26] UCL, Dept Comp Sci, London W1W 7TS, England

[27] Res Studios Austria FG, Data Sci Studio, A-1090 Vienna, Austria

[28] Eindhoven Univ Technol, Dept Elect Engn, NL-5600 MB Eindhoven, Netherlands

[29] Natl Taiwan Univ Sci & Technol, Grad Inst Biomed Engn, AIExplore, NTUST Ctr Comp Vis & Med Imaging, Taipei 106, Taiwan

[30] Univ Med Ctr Rostock, Inst Diagnost & Intervent Radiol, D-18051 Rostock, Germany

[31] Univ Bern, Inst Surg Technol & Biomech, CH-3014 Bern, Switzerland

[32] Univ Rennes, INSERM, LTSI UMR S 1099, F-35043 Rennes, France

[33] German Canc Res Ctr, Div Biostat, D-69120 Heidelberg, Germany

来源：

NATURE COMMUNICATIONS | 2018年 / 9卷

基金：

澳大利亚研究理事会; 欧洲研究理事会; 英国医学研究理事会; 英国工程与自然科学研究理事会; 英国惠康基金; 瑞士国家科学基金会;

关键词：

COMPUTED-TOMOGRAPHY SCANS; SEGMENTATION ALGORITHMS; STANDARDIZED EVALUATION; EVALUATION FRAMEWORK; LESION SEGMENTATION; LUMEN SEGMENTATION; PULMONARY NODULES; VALIDATION; TRACKING; MRI;

D O I：

10.1038/s41467-018-07619-7

中图分类号：

O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];

学科分类号：

07 ; 0710 ; 09 ;

摘要：

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

引用

页数：13

共 65 条

[11] Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results From the MICCAI 2015 Endoscopic Vision Challenge [J].

Bernal, Jorge ;

Tajkbaksh, Nima ;

Sanchez, Francisco Javier ;

Matuszewski, Bogdan J. ;

Chen, Hao ;

Yu, Lequan ;

Angermann, Quentin ;

Romain, Olivier ;

Rustad, Bjorn ;

Balasingham, Ilangko ;

Pogorelov, Konstantin ;

Choi, Sungbin ;

Debard, Quentin ;

Maier-Hein, Lena ;

Speidel, Stefanie ;

Stoyanov, Danail ;

Brandao, Patrick ;

Cordova, Henry ;

Sanchez-Montes, Cristina ;

Gurudu, Suryakanth R. ;

Fernandez-Esparrach, Gloria ;

Dray, Xavier ;

Liang, Jianming ;

Histace, Aymeric .

IEEE TRANSACTIONS ON MEDICAL IMAGING, 2017, 36 (06) :1231-1249

[12] Standardized Evaluation System for Left Ventricular Segmentation Algorithms in 3D Echocardiography [J].

Bernard, Olivier ;

Bosch, Johan G. ;

Heyde, Brecht ;

Alessandrini, Martino ;

Barbosa, Daniel ;

Camarasu-Pop, Sorina ;

Cervenansky, Frederic ;

Valette, Sebastien ;

Mirea, Oana ;

Bernier, Michel ;

Jodoin, Pierre-Marc ;

Domingos, Jaime Santo ;

Stebbing, Richard V. ;

Keraudren, Kevin ;

Oktay, Ozan ;

Caballero, Jose ;

Shi, Wei ;

Rueckert, Daniel ;

Milletari, Fausto ;

Ahmadi, Seyed-Ahmad ;

Smistad, Erik ;

Lindseth, Frank ;

van Stralen, Maartje ;

Wang, Chen ;

Smedby, Orjan ;

Donal, Erwan ;

Monaghan, Mark ;

Papachristidis, Alex ;

Geleijnse, Marcel L. ;

Galli, Elena ;

D'hooge, Jan .

IEEE TRANSACTIONS ON MEDICAL IMAGING, 2016, 35 (04) :967-977

[13] Toward better benchmarking: challenge-based methods assessment in cancer genomics [J].

Boutros, Paul C. ;

Margolin, Adam A. ;

Stuart, Joshua M. ;

Califano, Andrea ;

Stolovitzky, Gustavo .

GENOME BIOLOGY, 2014, 15 (09) :462

[14] Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: The CADDementia challenge [J].

Bron, Esther E. ;

Smits, Marion ;

van der Flier, Wiesje M. ;

Vrenken, Hugo ;

Barkhof, Frederik ;

Scheltens, Philip ;

Papma, Janne M. ;

Steketee, Rebecca M. E. ;

Orellana, Carolina Mendez ;

Meijboom, Rozanna ;

Pinto, Madalena ;

Meireles, Joana R. ;

Garrett, Carolina ;

Bastos-Leite, Antonio J. ;

Abdulkadir, Ahmed ;

Ronneberger, Olaf ;

Amoroso, Nicola ;

Bellotti, Roberto ;

Cardenas-Pena, David ;

Alvarez-Meza, Andres M. ;

Dolph, Chester V. ;

Iftekharuddin, Khan M. ;

Eskildsen, Simon F. ;

Coupe, Pierrick ;

Fonov, Vladimir S. ;

Franke, Katja ;

Gaser, Christian ;

Ledig, Christian ;

Guerrero, Ricardo ;

Tong, Tong ;

Gray, Katherine R. ;

Moradi, Elaheh ;

Tohka, Jussi ;

Routier, Alexandre ;

Durrleman, Stanley ;

Sarica, Alessia ;

Di Fatta, Giuseppe ;

Sensi, Francesco ;

Chincarini, Andrea ;

Smith, Garry M. ;

Stoyanov, Zhivko V. ;

Sorensen, Lauge ;

Nielsen, Mads ;

Tangaro, Sabina ;

Inglese, Paolo ;

Wachinger, Christian ;

Reuter, Martin ;

van Swieten, John C. ;

Niessen, Wiro J. ;

Klein, Stefan .

NEUROIMAGE, 2015, 111 :562-579

[15] Longitudinal multiple sclerosis lesion segmentation: Resource and challenge [J].

Carass, Aaron ;

Roy, Snehashis ;

Jog, Amod ;

Cuzzocreo, Jennifer L. ;

Magrath, Elizabeth ;

Gherman, Adrian ;

Button, Julia ;

Nguyen, James ;

Prados, Ferran ;

Sudre, Carole H. ;

Cardoso, Manuel Jorge ;

Cawley, Niamh ;

Ciccarelli, Olga ;

Wheeler-Kingshott, Claudia A. M. ;

Ourselin, Sebastien ;

Catanese, Laurence ;

Deshpande, Hrishikesh ;

Maurel, Pierre ;

Commowick, Olivier ;

Barillot, Christian ;

Tomas-Fernandez, Xavier ;

Warfield, Simon K. ;

Vaidya, Suthirth ;

Chunduru, Abhijith ;

Muthuganapathy, Ramanathan ;

Krishnamurthi, Ganapathy ;

Jesson, Andrew ;

Arbel, Tal ;

Maier, Oskar ;

Handeles, Heinz ;

Iheme, Leonardo O. ;

Unay, Devrim ;

Jain, Saurabh ;

Sima, Diana M. ;

Smeets, Dirk ;

Ghafoorian, Mohsen ;

Platel, Bram ;

Birenbaum, Ariel ;

Greenspan, Hayit ;

Bazin, Pierre-Louis ;

Calabresi, Peter A. ;

Crainiceanu, Ciprian M. ;

Ellingsen, Lotta M. ;

Reich, Daniel S. ;

Prince, Jerry L. ;

Pham, Dzung L. .

NEUROIMAGE, 2017, 148 :77-102

[16] A Review of Biological Image Analysis [J].

Chen, Weiyang ;

Li, Weiwei ;

Dong, Xiangjun ;

Pei, Jialun .

CURRENT BIOINFORMATICS, 2018, 13 (04) :337-343

[17] Objective comparison of particle tracking methods [J].

Chenouard, Nicolas ;

Smal, Ihor ;

de Chaumont, Fabrice ;

Maska, Martin ;

Sbalzarini, Ivo F. ;

Gong, Yuanhao ;

Cardinale, Janick ;

Carthel, Craig ;

Coraluppi, Stefano ;

Winter, Mark ;

Cohen, Andrew R. ;

Godinez, William J. ;

Rohr, Karl ;

Kalaidzidis, Yannis ;

Liang, Liang ;

Duncan, James ;

Shen, Hongying ;

Xu, Yingke ;

Magnusson, Klas E. G. ;

Jalden, Joakim ;

Blau, Helen M. ;

Paul-Gilloteaux, Perrine ;

Roudot, Philippe ;

Kervrann, Charles ;

Waharte, Francois ;

Tinevez, Jean-Yves ;

Shorte, Spencer L. ;

Willemse, Joost ;

Celler, Katherine ;

van Wezel, Gilles P. ;

Dan, Han-Wei ;

Tsai, Yuh-Show ;

Ortiz de Solorzano, Carlos ;

Olivo-Marin, Jean-Christophe ;

Meijering, Erik .

NATURE METHODS, 2014, 11 (03) :281-U247

[18] THE ASLIB CRANFIELD RESEARCH-PROJECT ON THE COMPARATIVE EFFICIENCY OF INDEXING SYSTEMS [J].

CLEVERDON, CW .

ASLIB PROCEEDINGS, 1960, 12 (12) :421-431

[19] Quantitative Comparison of Reconstruction Methods for Intra-Voxel Fiber Recovery From Diffusion MRI [J].

Daducci, Alessandro ;

Jorge Canales-Rodriguez, Erick ;

Descoteaux, Maxime ;

Garyfallidis, Eleftherios ;

Gur, Yaniv ;

Lin, Ying-Chia ;

Mani, Merry ;

Merlet, Sylvain ;

Paquette, Michael ;

Ramirez-Manzanares, Alonso ;

Reisert, Marco ;

Reis Rodrigues, Paulo ;

Sepehrband, Farshid ;

Caruyer, Emmanuel ;

Choupan, Jeiran ;

Deriche, Rachid ;

Jacob, Mathews ;

Menegaz, Gloria ;

Prckovska, Vesna ;

Rivera, Mariano ;

Wiaux, Yves ;

Thiran, Jean-Philippe .

IEEE TRANSACTIONS ON MEDICAL IMAGING, 2014, 33 (02) :384-399

[20] MEASURES OF THE AMOUNT OF ECOLOGIC ASSOCIATION BETWEEN SPECIES [J].

DICE, LR .

ECOLOGY, 1945, 26 (03) :297-302

← 1 2 3 4 5 6 7 →