Unsupervised visual similarity-based medical image retrieval via dual-encoder and metric learning

Skin cancer stands as one of the most lethal cancers, with its prognosis heavily reliant upon timely diagnosis. Medical image retrieval (MIR) techniques aim to retrieve images similar to the query one from a dataset, assisting doctors in disease diagnosis and clinical treatment programming. However, a significant challenge in dermatoscopic image interpretation stems from the heterogeneity of lesion visual appearance, which hinders existing methods from effectively balancing visual similarity with disease category. To address this issue, we propose an unsupervised approach for medical image retrieval to enhance the visual congruity of retrieval while ensuring disease accuracy.Dual-Encoder is trained to extract image features via a self-distilling dual network, and followed by subsequent refinement of the feature representation utilizing unsupervised metric learning methodologies. Our method is tested on the skin cancer dataset ISIC2019, evaluated using dual-dimensional metrics of visual and disease similarity. To assess visual similarity performance, we constructed a test visual dataset and categorize into 27 visual types we fine-tuned the pre-trained model on multiple datasets encompassing ultrasound, MRI and CT images to evaluate its applicability, which outperforms other advanced generalized retrieval methods. The experiments demostrated the effectiveness and generalization of our proposed method.