Novel Human Artificial Intelligence Hybrid Framework Pinpoints Thyroid Nodule Malignancy and Identifies Overlooked Second-Order Ultrasonographic Features

Simple Summary: Deep learning-based computer-aided diagnosis has gained momentum in the radiology field thanks to the technological advances of convolutional neural networks (CNN). However, how to utilize the black-box predictions of these CNN models to the clinical routine still relies on radiologists' personal judgements. In addition, existing CNN models only improve radiologists' diagnosis when they outperform the radiologists, thereby limiting their added values for possible efficiency enhancement and improving mostly the diagnostic performances of junior radiologists. We present a Human Artificial Intelligence Hybrid (HAIbrid) integrating framework that reweights Thyroid Imaging Reporting and Data System (TIRADS) features and the malignancy score predicted by a convolutional neural network (CNN) for nodule malignancy stratification and diagnosis. We defined extra ultrasonographical features from color Doppler images to explore malignancy-relevant features. We proposed Gated Attentional Factorization Machine (GAFM) to identify second-order interacting features trained via a 10 fold distribution-balanced stratified cross-validation scheme on ultrasound images of 3002 nodules all finally characterized by postoperative pathology (1270 malignant ones), retrospectively collected from 131 hospitals. Our GAFM-HAIbrid model demonstrated significant improvements in Area Under the Curve (AUC) value (p-value < 10(-5)), reaching about 0.92 over the standalone CNN (similar to 0.87) and senior radiologists (similar to 0.86), and identified a second-order vascularity localization and morphological pattern which was overlooked if only first-order features were considered. We validated the advantages of the integration framework on an already-trained commercial CNN system and our findings using an extra set of ultrasound images of 500 nodules. Our HAIbrid framework allows natural integration to clinical workflow for thyroid nodule malignancy risk stratification and diagnosis, and the proposed GAFM-HAIbrid model may help identify novel diagnosis-relevant second-order features beyond ultrasonography.