TC-Net: A joint learning framework based on CNN and vision transformer for multi-lesion medical images segmentation

Background: With the rapid advancement of medical imaging technology, the demand for accurate segmentation of medical images is increasing. However, most existing methods are unable to capture locality and long-range dependency information in integrated ways for medical images.Method: In this paper, we propose an elegant segmentation framework for medical images named TC-Net, which can utilize both the locality-aware and long-range dependencies in the medical images. As for the locality-aware perspective, we employ a CNN-based encoder and decoder structure. The CNN branch uses the locality of convolution operations to dig out local information in medical images. As for the long-range dependencies, we construct a Transformer branch to focus on the global context. Additionally, we proposed a locality-aware and long-range dependency concatenation strategy (LLCS) to aggregate the feature maps obtained from the two subbranches. Finally, we present a dynamic cyclical focal loss (DCFL) to address the class imbalance problem in multi-lesion segmentation.Results: Comprehensive experiments were conducted on lesion segmentation tasks using two fundus image da-tabases and a skin image database. The TC-Net achieves scores of 0.6985 and 0.5171 in the metric of mean pixel accuracy on the IDRiD and DDR databases, respectively. Moreover, on the skin image database, the TC-Net reached mean pixel accuracy of 0.8886. The experiment results demonstrate that the proposed method ach-ieves better performance than other deep learning segmentation schemes. Furthermore, the proposed DCFL achieves higher performance than other loss functions in multi-lesion segmentation.Significance: The proposed TC-Net is a promising new framework for multi-lesion medical image segmentation and many other challenging image segmentation tasks. (c) 2001 Elsevier Science. All rights reserved.