Using Crowdsourcing for Multi-label Biomedical Compound Figure Annotation

被引:1
|
作者
Seco de Herrera, Alba Garcia [1 ]
Schaer, Roger [2 ]
Antani, Sameer [1 ]
Mueller, Henning [2 ]
机构
[1] Natl Lib Med, Lister Hill Natl Ctr Biomed Commun, Bethesda, MD USA
[2] Univ Appl Sci Western Switzerland HES SO, Sierre, Switzerland
来源
DEEP LEARNING AND DATA LABELING FOR MEDICAL APPLICATIONS | 2016年 / 10008卷
关键词
Multi-label annotation; Compound figures; Crowdsourcing;
D O I
10.1007/978-3-319-46976-8_24
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Information analysis or retrieval for images in the biomedical literature needs to deal with a large amount of compound figures (figures containing several subfigures), as they constitute probably more than half of all images in repositories such as PubMed Central, which was the data set used for the task. The ImageCLEFmed benchmark proposed among other tasks in 2015 and 2016 a multi-label classification task, which aims at evaluating the automatic classification of figures into 30 image types. This task was based on compound figures and thus the figures were distributed to participants as compound figures but also in a separated form. Therefore, the generation of a gold standard was required, so that algorithms of participants can be evaluated and compared. This work presents the process carried out to generate the multi-labels of similar to 2650 compound figures using a crowdsourcing approach. Automatic algorithms to separate compound figures into subfigures were used and the results were then validated or corrected via crowdsourcing. The image types (MR, CT, X-ray,...) were also annotated by crowdsourcing including detailed quality control. Quality control is necessary to insure quality of the annotated data as much as possible. similar to 625 h were invested with a cost of similar to 870$.
引用
收藏
页码:228 / 237
页数:10
相关论文
共 50 条
  • [21] MULTI-LABEL IMAGE ANNOTATION VIA MAXIMUM CONSISTENCY
    Wang, Hua
    Hu, Jian
    2010 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, 2010, : 2337 - 2340
  • [22] Adaptive Graph Guided Embedding for Multi-label Annotation
    Wang, Lichen
    Ding, Zhengming
    Fu, Yun
    PROCEEDINGS OF THE TWENTY-SEVENTH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 2018, : 2798 - 2804
  • [23] Automatic image annotation using model fusion and multi-label selection algorithm
    Wang, Liqin
    Zhang, Aofan
    Wang, Peng
    Dong, Yongfeng
    JOURNAL OF INTELLIGENT & FUZZY SYSTEMS, 2019, 37 (04) : 4999 - 5008
  • [24] Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations
    Stureborg, Rickard
    Dhingra, Bhuwan
    Yang, Jun
    PROCEEDINGS OF THE 2023 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS (CHI 2023), 2023,
  • [25] Deep Multi-Instance Multi-Label Learning for Image Annotation
    Guo, Hai-Feng
    Han, Lixin
    Su, Shoubao
    Sun, Zhou-Bao
    INTERNATIONAL JOURNAL OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE, 2018, 32 (03)
  • [26] Multi-instance multi-label learning for surgical image annotation
    Loukas, Constantinos
    Sgouros, Nicholas P.
    INTERNATIONAL JOURNAL OF MEDICAL ROBOTICS AND COMPUTER ASSISTED SURGERY, 2020, 16 (02):
  • [27] Multi-view based multi-label propagation for image annotation
    He, Zhanying
    Chen, Chun
    Bu, Jiajun
    Li, Ping
    Cai, Deng
    NEUROCOMPUTING, 2015, 168 : 853 - 860
  • [28] Multi-label Annotation for Visual Multi-Task Learning Models
    Sharma, Gaurang
    Angleraud, Alexandre
    Pieters, Roel
    2023 SEVENTH IEEE INTERNATIONAL CONFERENCE ON ROBOTIC COMPUTING, IRC 2023, 2023, : 31 - 34
  • [29] Multi-Label Modality Classification for Figures in Biomedical Literature
    Lagopoulos, Athanasios
    Fachantidis, Anestis
    Tsoumakas, Grigorios
    2017 IEEE 30TH INTERNATIONAL SYMPOSIUM ON COMPUTER-BASED MEDICAL SYSTEMS (CBMS), 2017, : 79 - 84
  • [30] Research on Constructing a Multi-label Image Annotation and Retrieval System
    Tian, Yulong
    Li, Ran
    Lu, Jianjiang
    Zhang, YaFei
    Lu, Zining
    INFORMATION TECHNOLOGY FOR MANUFACTURING SYSTEMS, PTS 1 AND 2, 2010, : 559 - 564