Deep learning multimodal bilinear fusion network based on vibrational spectroscopy for diagnosis of benign and malignant thyroid tumors

Background: Thyroid tumors, a common neoplasm type, are classified into benign and malignant forms. Benign tumors include follicular adenoma and papillary cystadenoma, while malignant types cover papillary carcinoma, follicular carcinoma, medullary carcinoma, and anaplastic carcinoma. Accurate differentiation between benign and malignant thyroid tumors is essential for effective treatment and prognosis. In the era of big data, the complexity and variety of biological sample data have increased significantly, highlighting the potential of Raman and infrared spectroscopy for biological analysis. Deep learning techniques show superior performance over traditional machine learning in data preprocessing, feature extraction, spectral analysis, and highdimensional data modeling. Results: This study comprehensively examines the current research landscape and potential future directions for benign and malignant thyroid tumor diagnosis. We introduce a multimodal bilinear fusion network (MBFN) that integrates Raman and infrared spectroscopy data using deep learning for improved diagnostic accuracy. Our proposed MBFN architecture is based on convolutional neural networks (CNN) and bidirectional extended shortterm memory networks (BiLSTM), specifically designed to extract intra-modality spectral features while utilizing a bilinear fusion mechanism to enable interactive feature representation between modalities. Evaluated on a dataset comprising 196 thyroid tumor samples, the MBFN approach achieved an accuracy of 97.95 % and an area under the curve (AUC) of 97.97 %, surpassing conventional diagnostic approaches. This superior performance underscores the efficacy of MBFN in enhancing diagnostic accuracy, setting a new benchmark for thyroid tumor classification. Significance: The findings of this research contribute novel methodologies and perspectives for the early diagnosis of thyroid tumors. A deep learning multimodal fusion approach is particularly valuable for managing nonlinear data and integrating complementary information across modalities, thereby enhancing feature representation. The MBFN approach demonstrates the potential of multimodal deep learning frameworks in improving diagnostic accuracy and advancing thyroid tumor research, paving the way for broader applications in the biomedical diagnostic field.