Effect of Training Data Volume on Performance of Convolutional Neural Network Pneumothorax Classifiers

被引:14
作者
Thian, Yee Liang [1 ]
Ng, Dian Wen [1 ,2 ]
Hallinan, James Thomas Patrick Decourcy [1 ]
Jagmohan, Pooja [1 ]
Sia, Soon Yiew [1 ]
Mohamed, Jalila Sayed Adnan [1 ,3 ]
Quek, Swee Tian [1 ]
Feng, Mengling [2 ]
机构
[1] Natl Univ Singapore Hosp, Dept Diagnost Imaging, 5 Lower Kent Ridge Rd, Singapore 119074, Singapore
[2] Natl Univ Singapore, Saw Swee Hock Sch Publ Hlth, Sch Comp Sci, Yong Loo Lin Sch Med, 12 Sci Dr 2,10-01 Queenstown, Singapore 117549, Singapore
[3] Salmaniya Med Complex Rd 2904, Manama, Bahrain
关键词
Pneumothorax; Convolutional neural network; Deep learning; Volume; Dataset size;
D O I
10.1007/s10278-022-00594-y
中图分类号
R8 [特种医学]; R445 [影像诊断学];
学科分类号
1002 ; 100207 ; 1009 ;
摘要
Large datasets with high-quality labels required to train deep neural networks are challenging to obtain in the radiology domain. This work investigates the effect of training dataset size on the performance of deep learning classifiers, focusing on chest radiograph pneumothorax detection as a proxy visual task in the radiology domain. Two open-source datasets (ChestX-ray14 and CheXpert) comprising 291,454 images were merged and convolutional neural networks trained with stepwise increase in training dataset sizes. Model iterations at each dataset volume were evaluated on an external test set of 525 emergency department chest radiographs. Learning curve analysis was performed to fit the observed AUCs for all models generated. For all three network architectures tested, model AUCs and accuracy increased rapidly from 2 x 10(3) to 20 x 10(3) training samples, with more gradual increase until the maximum training dataset size of 291 x 10(3) images. AUCs for models trained with the maximum tested dataset size of 291 x 10(3) images were significantly higher than models trained with 20 x 10(3) images: ResNet-50: AUC(20k) = 0.86, AUC(291k) = 0.95, p < 0.001; DenseNet-121 AUC(20k) = 0.85, AUC(291k) = 0.93, p < 0.001; EfficientNet AUC(20k) = 0.92, AUC (291 k) = 0.98, p < 0.001. Our study established learning curves describing the relationship between dataset training size and model performance of deep learning convolutional neural networks applied to a typical radiology binary classification task. These curves suggest a point of diminishing performance returns for increasing training data volumes, which algorithm developers should consider given the high costs of obtaining and labelling radiology data.
引用
收藏
页码:881 / 892
页数:12
相关论文
共 26 条
  • [1] Is your dataset big enough? Sample size requirements when using artificial neural networks for discrete choice analysis
    Alwosheel, Ahmad
    van Cranenburgh, Sander
    Chorus, Caspar G.
    [J]. JOURNAL OF CHOICE MODELLING, 2018, 28 : 167 - 182
  • [2] [Anonymous], 2015, How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?
  • [3] Sample-Size Determination Methodologies for Machine Learning in Medical Imaging Research: A Systematic Review
    Balki, Indranil
    Amirabadi, Afsaneh
    Levman, Jacob
    Martel, Anne L.
    Emersic, Ziga
    Meden, Blaz
    Garcia-Pedrero, Angel
    Ramirez, Saul C.
    Kong, Dehan
    Moody, Alan R.
    Tyrrell, Pascal N.
    [J]. CANADIAN ASSOCIATION OF RADIOLOGISTS JOURNAL-JOURNAL DE L ASSOCIATION CANADIENNE DES RADIOLOGISTES, 2019, 70 (04): : 344 - 353
  • [4] Deng J, 2009, PROC CVPR IEEE, P248, DOI 10.1109/CVPRW.2009.5206848
  • [5] Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs
    Dunnmon, Jared A.
    Yi, Darvin
    Langlotz, Curtis P.
    Re, Christopher
    Rubin, Daniel L.
    Lungren, Matthew P.
    [J]. RADIOLOGY, 2019, 290 (02) : 537 - 544
  • [6] Predicting sample size required for classification performance
    Figueroa, Rosa L.
    Zeng-Treitler, Qing
    Kandula, Sasikiran
    Ngo, Long H.
    [J]. BMC MEDICAL INFORMATICS AND DECISION MAKING, 2012, 12
  • [7] Deep Residual Learning for Image Recognition
    He, Kaiming
    Zhang, Xiangyu
    Ren, Shaoqing
    Sun, Jian
    [J]. 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, : 770 - 778
  • [8] Hestness J., 2017, PREPRINT
  • [9] Huang G., 2017, P IEEE C COMPUTER VI, P4700, DOI [DOI 10.1109/CVPR.2017.243, 10.48550/ARXIV.1608.06993]
  • [10] Irvin J, 2019, AAAI CONF ARTIF INTE, P590