Existing Transformer-based algorithms for single image deraining achieve state-of-the-art performance but leading to reasonable computational loads while failing to effectively process real-world rainy images. To this end, we propose a frequency-guided dual sparse self-attention Transformer for single image deraining (FDSATFormer). Initially, our proposed method utilizes the spatial sparse factor and the channel reduction factor to extract accurate global information and significantly decreases the amount of computation. Furthermore, we present dual sparse self-attention feature learning network (DSFL) to deal with the problem that Transformer is difficult to represent self-attention on high-resolution feature maps. Meanwhile, by exploring the spectral changes of rainy image before and after removing rain streaks, we develop a frequency-guided feature enhancement module (FFE), which exploits the global information from the frequency domain to guide the accurate learning of spatial features in network encoders. In addition, the encoder and decoder of most existing methods follow the similar principles, resulting in almost double computational burden. To handle with this issue, we propose a hierarchical feature decoding and reconstructing network (HFDR), which uses non-parametric spatial neighborhood shift (SNS) to construct the feature decoding network and achieves fine results while further reducing the overall computing burden. Experimental results show that, our method improves the average peak signal noise ratio by 3.13 dB and 0.12 dB, and achieves performance gains of 1.39% and 1.07% in average structure similarity over the state-of-the-art Uformer and Restormer. © 2023 Chinese Institute of Electronics. All rights reserved.
