Large-Scale Hierarchical Medical Image Retrieval Based on a Multilevel Convolutional Neural Network

Presently, with advancements in medical imaging modalities, various imaging methods are widely used in clinics. To efficiently assess and manage the images, in this paper, a content-based medical image retrieval (CBMIR) system is suggested as a clinical tool. A global medical image database is established through a collection of data from more than ten countries and dozens of sources, schools and laboratories. The database has more than 536 294 medical images, including 14 imaging modalities, 40 organs and 52 diseases. A multilevel convolutional neural network (MLCNN) using hierarchical progressive feature learning is subsequently proposed to perform hierarchical medical image retrieval, including multiple levels of image modalities, organs and diseases. At each classification level, a dense block is trained through a labeled classification. With the epochs increasing, four training stages are performed to simultaneously train the three levels with different weights of the loss function. Then, the trained features are used in the CBMIR system. The results show that using the MLCNN on a representative dataset can achieve a mAP of 0.86, which is higher than the 0.71 achieved by ResNet152 in the literature. Applying the hierarchical progressive feature learning can achieve a 12%-16% performance improvement in CNNs and outperform vision Transformer with only 63% of the training time. The proposed representative image selection and multilevel architecture improves the efficiency and precision of retrieving large-scale medical image databases.