Using Crowdsourcing for Multi-label Biomedical Compound Figure Annotation

被引:1
|
作者
Seco de Herrera, Alba Garcia [1 ]
Schaer, Roger [2 ]
Antani, Sameer [1 ]
Mueller, Henning [2 ]
机构
[1] Natl Lib Med, Lister Hill Natl Ctr Biomed Commun, Bethesda, MD USA
[2] Univ Appl Sci Western Switzerland HES SO, Sierre, Switzerland
来源
DEEP LEARNING AND DATA LABELING FOR MEDICAL APPLICATIONS | 2016年 / 10008卷
关键词
Multi-label annotation; Compound figures; Crowdsourcing;
D O I
10.1007/978-3-319-46976-8_24
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Information analysis or retrieval for images in the biomedical literature needs to deal with a large amount of compound figures (figures containing several subfigures), as they constitute probably more than half of all images in repositories such as PubMed Central, which was the data set used for the task. The ImageCLEFmed benchmark proposed among other tasks in 2015 and 2016 a multi-label classification task, which aims at evaluating the automatic classification of figures into 30 image types. This task was based on compound figures and thus the figures were distributed to participants as compound figures but also in a separated form. Therefore, the generation of a gold standard was required, so that algorithms of participants can be evaluated and compared. This work presents the process carried out to generate the multi-labels of similar to 2650 compound figures using a crowdsourcing approach. Automatic algorithms to separate compound figures into subfigures were used and the results were then validated or corrected via crowdsourcing. The image types (MR, CT, X-ray,...) were also annotated by crowdsourcing including detailed quality control. Quality control is necessary to insure quality of the annotated data as much as possible. similar to 625 h were invested with a cost of similar to 870$.
引用
收藏
页码:228 / 237
页数:10
相关论文
共 50 条
  • [31] Clustering Based Multi-Label Classification for Image Annotation and Retrieval
    Nasierding, Gulisong
    Tsoumakas, Grigorios
    Kouzani, Abbas Z.
    2009 IEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN AND CYBERNETICS (SMC 2009), VOLS 1-9, 2009, : 4514 - +
  • [32] MULTI-LABEL ANNOTATION STUDY IN VIDEO SEMANTIC CONTENT ANALYSIS
    Wei, Wei
    Liu, Wen-Qing
    Li, Li-Li
    PROCEEDINGS OF 2009 INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND CYBERNETICS, VOLS 1-6, 2009, : 2453 - 2457
  • [33] An Empirical Study of Multi-Label Learning Methods for Video Annotation
    Dimou, Anastasios
    Tsoumakas, Grigorios
    Mezaris, Vasileios
    Kompatsiaris, Ioannis
    Vlahavas, Ioannis
    CBMI: 2009 INTERNATIONAL WORKSHOP ON CONTENT-BASED MULTIMEDIA INDEXING, 2009, : 19 - +
  • [34] Content and Context-Based Multi-Label Image Annotation
    Lu, Hong
    Zheng, Yingbin
    Xue, Xiangyang
    Zhang, Yuejie
    2009 IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPR WORKSHOPS 2009), VOLS 1 AND 2, 2009, : 958 - 965
  • [35] Joint Multi-Label Attention Networks for Social Text Annotation
    Dong, Hang
    Wang, Wei
    Huang, Kaizhu
    Coenen, Frans
    2019 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES (NAACL HLT 2019), VOL. 1, 2019, : 1348 - 1354
  • [36] An unsupervised annotation of Arabic texts using multi-label topic modeling and genetic algorithm
    Almuzaini, Huda A.
    Azmi, Aqil M.
    EXPERT SYSTEMS WITH APPLICATIONS, 2022, 203
  • [37] A Multi-Instance Multi-Label Learning Approach for Protein Domain Annotation
    Meng, Yang
    Deng, Lei
    Chen, Zhigang
    Zhou, Cheng
    Liu, Diwei
    Fan, Chao
    Yan, Ting
    INTELLIGENT COMPUTING IN BIOINFORMATICS, 2014, 8590 : 104 - 111
  • [38] SIMULTANEOUS INSTANCE ANNOTATION AND CLUSTERING IN MULTI-INSTANCE MULTI-LABEL LEARNING
    Pham, Anh T.
    Raich, Raviv
    Fern, Xiaoli Z.
    2015 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, 2015,
  • [39] Dynamic Programming for Instance Annotation in Multi-Instance Multi-Label Learning
    Pham, Anh T.
    Raich, Raviv
    Fern, Xiaoli Z.
    IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2017, 39 (12) : 2381 - 2394
  • [40] INTEGRATING DISTANCE METRIC LEARNING INTO LABEL PROPAGATION MODEL FOR MULTI-LABEL IMAGE ANNOTATION
    Wang, Bin
    Shen, Yi
    Liu, Yuncai
    2011 18TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2011,