ATFormer: Advanced transformer for medical image segmentation

Combining transformers and convolutional neural networks is considered one of the most important directions for tackling medical image segmentation problems. To learn the long-range dependencies and local contexts, previous approaches embedded a convolutional layer into feedforward neural network inside the transformer block. However, a common issue is the instability during training since large differences in amplitude across layers by pre-layer normalization. Furthermore, multi-scale features were directly fused using the transformer from the encoder to decoder, which could disrupt valuable information for segmentation. To address these concerns, we propose Advanced TransFormer (ATFormer), a novel hybrid architecture that combines convo-lutional neural networks and transformers for medical image segmentation. First, the traditional transformer block has been refined into an Advanced Transformer Block, which adopts post-layer normalization to obtain mild activation values and employs the scaled cosine attention with shifted window for accurate spatial information. Second, the Progressive Guided Fusion module is introduced to make multi-scale features more discriminative while reducing the computational complexity. Experimental results on the ACDC, COVID-19 CT-Seg, and Tumor datasets demonstrate the significant advantage of ATFormer over existing methods that rely solely on convolutional neural networks, transformers, or their combination.