Monitoring Variations in the Use of Automated Contouring Software

被引:2
作者
Nealon, Kelly A. [1 ,2 ]
Han, Eun Young [3 ]
Kry, Stephen F. [4 ]
Nguyen, Callistus [5 ]
Pham, Mary [5 ]
Reed, Valerie K. [6 ]
Rosenthal, David [6 ]
Simiele, Samantha [3 ]
Court, Laurence E. [3 ]
机构
[1] Massachusetts Gen Hosp, Dept Radiat Oncol, Boston, MA 02114 USA
[2] Harvard Med Sch, Boston, MA 02115 USA
[3] Univ Texas MD Anderson Canc Ctr, Dept Radiat Phys Patient Care, Houston, TX USA
[4] Univ Texas MD Anderson Canc Ctr, Radiat Phys Outreach, Houston, TX USA
[5] Univ Texas MD Anderson Canc Ctr, Dept Radiat Phys Res, Houston, TX USA
[6] Univ Texas MD Anderson Canc Ctr, Dept Radiat Oncol, Houston, TX USA
基金
英国惠康基金;
关键词
CLINICAL-EVALUATION; SEGMENTATION; ACCURACY;
D O I
10.1016/j.prro.2023.09.004
中图分类号
R73 [肿瘤学];
学科分类号
100214 ;
摘要
Purpose: Our purpose was to identify variations in the clinical use of automatically generated contours that could be attributed to software error, off -label use, or automation bias. Methods and Materials: For 500 head and neck patients who were contoured by an in-house automated contouring system, Dice similarity coefficient and added path length were calculated between the contours generated by the automated system and the final contours after editing for clinical use. Statistical process control was used and control charts were generated with control limits at 3 standard deviations. Contours that exceeded the thresholds were investigated to determine the cause. Moving mean control plots were then generated to identify dosimetrists who were editing less over time, which could be indicative of automation bias. Results: Major contouring edits were flagged for: 1.0% brain, 3.1% brain stem, 3.5% left cochlea, 2.9% right cochlea, 4.8% esophagus, 4.1% left eye, 4.0% right eye, 2.2% left lens, 4.9% right lens, 2.5% mandible, 11% left optic nerve, 6.1% right optic nerve, 3.8% left parotid, 5.9% right parotid, and 3.0% of spinal cord contours. Identified causes of editing included unexpected patient positioning, deviation from standard clinical practice, and disagreement between dosimetrist preference and automated contouring style. A statistically significant (P < .05) difference was identified between the contour editing practice of dosimetrists, with 1 dosimetrist editing more across all organs at risk. Eighteen percent (27/150) of moving mean control plots created for 5 dosimetrists indicated the amount of contour editing was decreasing over time, possibly corresponding to automation bias. Conclusions: The developed system was used to detect statistically significant edits caused by software error, unexpected clinical use, and automation bias. The increased ability to detect systematic errors that occur when editing automatically generated contours will improve the safety of the automatic treatment planning workflow. (c) 2023 The Authors. Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
引用
收藏
页码:e75 / e85
页数:11
相关论文
共 50 条
[41]   Automated classification of software issue reports using machine learning techniques: an empirical study [J].
Pandey N. ;
Sanyal D.K. ;
Hudait A. ;
Sen A. .
Innovations in Systems and Software Engineering, 2017, 13 (04) :279-297
[42]   Automated software for PET/CT image registration to avoid unnecessary invasive cardiac surgery [J].
Khurshid, Khawar ;
Berger, Kevin ;
Khurshid, Khurram ;
McGough, Robert J. .
10TH IEEE INTERNATIONAL MULTITOPIC CONFERENCE 2006, PROCEEDINGS, 2006, :498-+
[43]   Sjogren-Lars son syndrome brain volumetric reductions demonstrated with an automated software [J].
de Castro, Jose Thiago de Souza ;
Saab, Camilo Lot fi ;
Souto, Mariam Patricia Auada ;
Ortolam, Juliane Giselle ;
Steiner, Carlos Eduardo ;
de Rezende, Thiago Junqueira Ribeiro ;
Reis, Fabiano .
ARQUIVOS DE NEURO-PSIQUIATRIA, 2023, 81 (09) :809-815
[44]   Novel fast semi-automated software to segment cartilage for knee MR acquisitions [J].
Duryea, J. ;
Neumann, G. ;
Brem, M. H. ;
Koh, W. ;
Noorbakhsh, F. ;
Jackson, R. D. ;
Yu, J. ;
Eaton, C. B. ;
Lang, P. .
OSTEOARTHRITIS AND CARTILAGE, 2007, 15 (05) :487-492
[45]   Automated quantification of dopaminergic immunostained neurons in substantia nigra using freely available software [J].
Bonaccorso Marinelli, Maria Paula ;
Baiardi, Gustavo ;
Ruth Valdez, Susana ;
Jorge Cabrera, Ricardo .
MEDICAL & BIOLOGICAL ENGINEERING & COMPUTING, 2022, 60 (10) :2995-3007
[46]   Developing an Automated Model-Based Software Testing Tool From the Design Phase [J].
Alsarraj, Rasha Gh. ;
Altaie, Atica M. ;
Zuhair Majeed, Esra .
IEEE ACCESS, 2025, 13 :58548-58558
[47]   Magnetic resonance image segmentation using semi-automated software for quantification of knee articular cartilage-initial evaluation of a technique for paired scans [J].
Brem, M. H. ;
Lang, P. K. ;
Neumann, G. ;
Schlechtweg, P. M. ;
Schneider, E. ;
Jackson, R. ;
Yu, J. ;
Eaton, C. B. ;
Hennig, F. F. ;
Yoshioka, H. ;
Pappas, G. ;
Duryea, J. .
SKELETAL RADIOLOGY, 2009, 38 (05) :505-511
[48]   Creation of RTOG compliant patient CT-atlases for automated atlas based contouring of local regional breast and high-risk prostate cancers [J].
Velker, Vikram M. ;
Rodrigues, George B. ;
Dinniwell, Robert ;
Hwee, Jeremiah ;
Louie, Alexander V. .
RADIATION ONCOLOGY, 2013, 8
[49]   Automated CT Segmentation for Rapid Assessment of Anatomical Variations in Head-and-Neck Radiation Therapy [J].
Dai, Xianjin ;
Lei, Yang ;
Wang, Tonghe ;
Tian, Zhen ;
Zhou, Jun ;
McDonald, Mark ;
Yu, David S. ;
Ghavidel, Beth B. ;
Bradley, Jeffrey D. ;
Liu, Tian ;
Yang, Xiaofeng .
MEDICAL IMAGING 2022: IMAGE-GUIDED PROCEDURES, ROBOTIC INTERVENTIONS, AND MODELING, 2022, 12034
[50]   Validation of a Magnetic Resonance Imaging-based Auto-contouring Software Tool for Gross Tumour Delineation in Head and Neck Cancer Radiotherapy Planning [J].
Doshi, T. ;
Wilson, C. ;
Paterson, C. ;
Lamb, C. ;
James, A. ;
MacKenzie, K. ;
Soraghan, J. ;
Petropoulakis, L. ;
Di Caterina, G. ;
Grose, D. .
CLINICAL ONCOLOGY, 2017, 29 (01) :60-67