Assessment of the Diagnostic Performance of a Commercially Available Artificial Intelligence Algorithm for Risk Stratification of Thyroid Nodules on Ultrasound

被引:0
作者
Ashton, Jeffrey [1 ]
Morrison, Samantha [2 ]
Erkanli, Alaattin [2 ]
Wildman-Tobriner, Benjamin [1 ]
机构
[1] Duke Univ, Med Ctr, Dept Radiol, 2301 ERWIN RD,Box 3808, Durham, NC 27710 USA
[2] Duke Univ, Sch Med, Dept Biostat & Bioinformat, Durham, NC USA
基金
美国国家卫生研究院;
关键词
AI; neoplasm; ultrasound; algorithm; koios; AMERICAN-COLLEGE; CANCER;
D O I
10.1089/thy.2024.0410
中图分类号
R5 [内科学];
学科分类号
1002 ; 100201 ;
摘要
Background: Thyroid nodules are challenging to accurately characterize on ultrasound (US), though the emergence of risk stratification systems and more recently artificial intelligence (AI) algorithms has improved nodule classification. The purpose of this study was to evaluate the performance of a recent Food and Drug Administration (FDA)-cleared AI tool for detection of malignancy in thyroid nodules on US. Methods: One year of consecutive thyroid US with >= 1 nodule from Duke University Hospital and its affiliate community hospital (649 nodules from 347 patients) were retrospectively evaluated. Included nodules had ground truth diagnoses by surgical pathology, fine needle aspiration (FNA), or three-year follow-up US showing stability. An FDA-cleared AI tool (Koios DS Thyroid) analyzed each nodule to generate (i) American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) descriptors, scores, and follow-up recommendations and (ii) an AI-adapter score to further adjust risk assessments and recommendations. Four groups were then compared: (i) Koios with AI-adapter, (ii) Koios without AI-adapter, (iii) clinical radiology report, and (iv) radiology report combined with AI-adapter. Performance of the final recommendations (FNA or no FNA) was determined based on ground truth, and comparison between the four groups was made using sensitivity, specificity, and receiver-operating-curve analysis. Results: Of 649 nodules, 32 were malignant and 617 were benign. Performance of Koios with AI-adapter enabled was similar to radiologists (area under the curve [AUC] 0.70 for both, [CI 0.60-0.81] and [0.60-0.79], respectively). Koios with AI-adapter had improved specificity compared to radiologists (0.63 [CI: 0.59-0.67] versus 0.43 [CI: 0.38-0.48]) but decreased sensitivity (0.69 [CI: 0.50-0.83) versus 0.81 [CI: 0.61, 0.92]). Highest performance was seen when the radiology interpretation was combined with the AI-adapter (AUC 0.76, [CI: 0.67-0.85]). Combined with the AI-adapter, radiologist specificity improved from 0.43 ([CI: 0.38-0.48]) to 0.53 ([CI: 0.49-0.58]) (McNemar's test p < 0.001), resulting in 17% fewer FNA recommendations, with unchanged sensitivity (0.81, p = 1). Conclusion: Koios DS demonstrated standalone performance similar to radiologists, though with lower sensitivity and higher specificity. Performance was best when radiologist interpretations were combined with the AI-adapter component, with improved specificity and reduced unnecessary FNA recommendations.
引用
收藏
页码:1379 / 1388
页数:10
相关论文
共 29 条
[1]   Improving the Efficacy of ACR TI-RADS Through Deep Learning-Based Descriptor Augmentation [J].
Barinov, Lev ;
Jairaj, Ajit ;
Middleton, William D. ;
Beland, Michael D. ;
Kirsch, Jonathan ;
Filice, Ross W. ;
Reverter, Jordi L. ;
Arguelles, Inaki ;
Grant, Edward G. .
JOURNAL OF DIGITAL IMAGING, 2023, 36 (06) :2392-2401
[2]   Ultrasonic S-Detect mode for the evaluation of thyroid nodules: A meta-analysis [J].
Bian, Jinyi ;
Wang, Ruyue ;
Lin, Mingxin .
MEDICINE, 2022, 101 (34) :E29991
[3]   Artificial Intelligence in Thyroid Field-A Comprehensive Review [J].
Bini, Fabiano ;
Pica, Andrada ;
Azzimonti, Laura ;
Giusti, Alessandro ;
Ruinelli, Lorenzo ;
Marinozzi, Franco ;
Trimboli, Pierpaolo .
CANCERS, 2021, 13 (19)
[4]   Artificial intelligence in thyroid ultrasound [J].
Cao, Chun-Li ;
Li, Qiao-Li ;
Tong, Jin ;
Shi, Li-Nan ;
Li, Wen-Xiao ;
Xu, Ya ;
Cheng, Jing ;
Du, Ting-Ting ;
Li, Jun ;
Cui, Xin-Wu .
FRONTIERS IN ONCOLOGY, 2023, 13
[5]   Diagnostic performance evaluation of different TI-RADS using ultrasound computer-aided diagnosis of thyroid nodules: An experience with adjusted settings [J].
Chambara, Nonhlanhla ;
Liu, Shirley Y. W. ;
Lo, Xina ;
Ying, Michael .
PLOS ONE, 2021, 16 (01)
[6]   Summary of meta-analyses of studies involving TIRADS classifications (EU-TIRADS, ACR-TIRADS, and K-TIRADS) in evaluating the malignant potential of focal lesions of the thyroid gland [J].
Dobruch-Sobczak, Katarzyna ;
Adamczewski, Zbigniew ;
Dedecjus, Marek ;
Lewinski, Andrzej ;
Migda, Bartosz ;
Ruchala, Marek ;
Skowronska-Szczesniak, Anna ;
Szczepanek-Parulska, Ewelina ;
Zajkowska, Klaudia ;
Zylka, Agnieszka .
JOURNAL OF ULTRASONOGRAPHY, 2022, 22 (89) :121-129
[7]   Contemporary Thyroid Nodule Evaluation and Management [J].
Grani, Giorgio ;
Sponziello, Marialuisa ;
Pecce, Valeria ;
Ramundo, Valeria ;
Durante, Cosimo .
JOURNAL OF CLINICAL ENDOCRINOLOGY & METABOLISM, 2020, 105 (09) :2869-2883
[8]   Applications of machine learning and deep learning to thyroid imaging: where do we stand? [J].
Ha, Eun Ju ;
Baek, Jung Hwan .
ULTRASONOGRAPHY, 2021, 40 (01) :23-29
[9]   Computer-Aided Diagnostic System for Thyroid Nodules on Ultrasonography: Diagnostic Performance Based on the Thyroid Imaging Reporting and Data System Classification and Dichotomous Outcomes [J].
Han, M. ;
Ha, E. J. ;
Park, J. H. .
AMERICAN JOURNAL OF NEURORADIOLOGY, 2021, 42 (03) :559-565
[10]   2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer [J].
Haugen, Bryan R. ;
Alexander, Erik K. ;
Bible, Keith C. ;
Doherty, Gerard M. ;
Mandel, Susan J. ;
Nikiforov, Yuri E. ;
Pacini, Furio ;
Randolph, Gregory W. ;
Sawka, Anna M. ;
Schlumberger, Martin ;
Schuff, Kathryn G. ;
Sherman, Steven I. ;
Sosa, Julie Ann ;
Steward, David L. ;
Tuttle, R. Michael ;
Wartofsky, Leonard .
THYROID, 2016, 26 (01) :1-133