On the usage of average Hausdorff distance for segmentation performance assessment: hidden error when used for ranking

被引:96
作者
Aydin, Orhun Utku [1 ]
Taha, Abdel Aziz [2 ]
Hilbert, Adam [1 ]
Khalil, Ahmed A. [3 ,4 ,5 ]
Galinovic, Ivana [3 ]
Fiebach, Jochen B. [3 ]
Frey, Dietmar [1 ]
Madai, Vince Istvan [1 ,6 ]
机构
[1] Charite Univ Med Berlin, CLAIM Charite Lab Artificial Intelligence Med, Berlin, Germany
[2] Res Studios Austria, Res Studio Data Sci, Salzburg, Austria
[3] Charite Univ Med Berlin, Ctr Stroke Res Berlin, Berlin, Germany
[4] Max Planck Inst Human Cognit & Brain Sci, Dept Neurol, Leipzig, Germany
[5] Humboldt Univ, Mind Brain Body Inst, Berlin Sch Mind & Brain, Berlin, Germany
[6] Birmingham City Univ, Fac Comp Engn & Built Environm, Sch Comp & Digital Technol, Birmingham, W Midlands, England
关键词
Average Hausdorff distance; Cerebral angiography; Cerebral arteries; Image processing (computer-assisted);
D O I
10.1186/s41747-020-00200-2
中图分类号
R8 [特种医学]; R445 [影像诊断学];
学科分类号
1002 ; 100207 ; 1009 ;
摘要
Average Hausdorff distance is a widely used performance measure to calculate the distance between two point sets. In medical image segmentation, it is used to compare ground truth images with segmentations allowing their ranking. We identified, however, ranking errors of average Hausdorff distance making it less suitable for applications in segmentation performance assessment. To mitigate this error, we present a modified calculation of this performance measure that we have coined "balanced average Hausdorff distance". To simulate segmentations for ranking, we manually created non-overlapping segmentation errors common in magnetic resonance angiography cerebral vessel segmentation as our use-case. Adding the created errors consecutively and randomly to the ground truth, we created sets of simulated segmentations with increasing number of errors. Each set of simulated segmentations was ranked using both performance measures. We calculated the Kendall rank correlation coefficient between the segmentation ranking and the number of errors in each simulated segmentation. The rankings produced by balanced average Hausdorff distance had a significantly higher median correlation (1.00) than those by average Hausdorff distance (0.89). In 200 total rankings, the former misranked 52 whilst the latter misranked 179 segmentations. Balanced average Hausdorff distance is more suitable for rankings and quality assessment of segmentations than average Hausdorff distance.
引用
收藏
页数:7
相关论文
共 12 条
[1]   Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing [J].
AlBadawy, Ehab A. ;
Saha, Ashirbani ;
Mazurowski, Maciej A. .
MEDICAL PHYSICS, 2018, 45 (03) :1150-1158
[2]   Tracking Pylorus in Ultrasonic Image Sequences With Edge-Based Optical Flow [J].
Chen, Chaojie ;
Wang, Yuanyuan ;
Yu, Jinhua ;
Zhou, Zhuyu ;
Shen, Li ;
Chen, Yaqing .
IEEE TRANSACTIONS ON MEDICAL IMAGING, 2012, 31 (03) :843-855
[3]   MR Imaging of the Extracranial Facial Nerve with the CISS Sequence [J].
Guenette, J. P. ;
Ben-Shlomo, N. ;
Jayender, J. ;
Seethamraju, R. T. ;
Kimbrell, V ;
Tran, N-A ;
Huang, R. Y. ;
Kim, C. J. ;
Kass, J., I ;
Corrales, C. E. ;
Lee, T. C. .
AMERICAN JOURNAL OF NEURORADIOLOGY, 2019, 40 (11) :1954-1959
[4]   BRAVE-NET: Fully Automated Arterial Brain Vessel Segmentation in Patients With Cerebrovascular Disease [J].
Hilbert, Adam ;
Madai, Vince I. ;
Akay, Ela M. ;
Aydin, Orhun U. ;
Behland, Jonas ;
Sobesky, Jan ;
Galinovic, Ivana ;
Khalil, Ahmed A. ;
Taha, Abdel A. ;
Wuerfel, Jens ;
Dusek, Petr ;
Niendorf, Thoralf ;
Fiebach, Jochen B. ;
Frey, Dietmar ;
Livne, Michelle .
FRONTIERS IN ARTIFICIAL INTELLIGENCE, 2020, 3
[5]   Prospective study on the mismatch concept in acute stroke patients within the first 24 h after symptom onset-1000Plus study [J].
Hotter, Benjamin ;
Pittl, Sandra ;
Ebinger, Martin ;
Oepen, Gabriele ;
Jegzentis, Kati ;
Kudo, Kohsuke ;
Rozanski, Michal ;
Schmidt, Wolf U. ;
Brunecker, Peter ;
Xu, Chao ;
Martus, Peter ;
Endres, Matthias ;
Jungehuelsing, Gerhard J. ;
Villringer, Arno ;
Fiebach, Jochen B. .
BMC NEUROLOGY, 2009, 9
[6]   Manual prostate cancer segmentation in MRI: interreader agreement and volumetric correlation with transperineal template core needle biopsy [J].
Liechti, Marc R. ;
Muehlematter, Urs J. ;
Schneider, Aurelia F. ;
Eberli, Daniel ;
Rupp, Niels J. ;
Hotker, Andreas M. ;
Donati, Olivio F. ;
Becker, Anton S. .
EUROPEAN RADIOLOGY, 2020, 30 (09) :4806-4815
[7]   A U-Net Deep Learning Framework for High Performance Vessel Segmentation in Patients With Cerebrovascular Disease [J].
Livne, Michelle ;
Rieger, Jana ;
Aydin, Orhun Utku ;
Taha, Abdel Aziz ;
Akay, Ela Marie ;
Kossen, Tabea ;
Sobesky, Jan ;
Kelleher, John D. ;
Hildebrand, Kristian ;
Frey, Dietmar ;
Madai, Vince, I .
FRONTIERS IN NEUROSCIENCE, 2019, 13
[8]   Tumor volume delineation us ing non-EPI diffusion weighted MRI and FDG-PET in he ad-and-neck patients. [J].
Peltenburg, B. ;
Schakel, T. ;
Dankbaar, J. W. ;
Aristophanous, M. ;
Terhaard, C. H. J. ;
Hoogduin, J. M. ;
Philippens, M. E. P. .
RADIOTHERAPY AND ONCOLOGY, 2017, 123 :S496-S497
[9]   Atlas-Based Segmentation of Temporal Bone Anatomy [J].
Powell, Kimerly A. ;
Liang, Tong ;
Hittle, Brad ;
Stredney, Don ;
Kerwin, Thomas ;
Wiet, Gregory J. .
INTERNATIONAL JOURNAL OF COMPUTER ASSISTED RADIOLOGY AND SURGERY, 2017, 12 (11) :1937-1944
[10]   Impact of inter-reader contouring variability on textural radiomics of colorectal liver metastases [J].
Rizzetto, Francesco ;
Calderoni, Francesca ;
De Mattia, Cristina ;
Defeudis, Arianna ;
Giannini, Valentina ;
Mazzetti, Simone ;
Vassallo, Lorenzo ;
Ghezzi, Silvia ;
Sartore-Bianchi, Andrea ;
Marsoni, Silvia ;
Siena, Salvatore ;
Regge, Daniele ;
Torresin, Alberto ;
Vanzulli, Angelo .
EUROPEAN RADIOLOGY EXPERIMENTAL, 2020, 4 (01)