Hierarchical Decoder with Parallel Transformer and CNN for Medical Image Segmentation

With the success of Transformers, hybrid Transformer and CNN methods gain considerable popularity in medical image segmentation. These methods utilize a hybrid architecture that combines Transformers and CNNs to fuse global and local information, supplemented by a pyramid structure to facilitate multi-scale interaction. However, they encounter two primary limitations: (i) Transformer struggle to capture complete global information due to the sliding window nature of the convolutional operator, and (ii) the pyramid structure within single decoder fails to provide sufficient multi-scale interaction necessary for restoring detailed features at higher levels. In this paper, we introduce the Hierarchical Decoder with Parallel Transformer and CNN (HiPar), a novel architecture designed to address these limitations. Firstly, we present a parallel structure of Transformer and CNN to maximize the capture of both global and local features. Subsequently, we propose a hierarchical decoder to model multi-scale information and progressively restore spatial details. Additionally, we incorporate lightweight components to enhance the efficiency of feature representation. Extensive experiments demonstrate that our HiPar achieves state-of-the-art results on three popular medical image segmentation benchmarks: Synapse, ACDC and GlaS.