Multi-Level Alignments for Compressed Video Super-Resolution

Due to the limited transmission bandwidth, to meet the application needs of consumer electronics products, there exists an approach to down-sample a video and then compress it to satisfy the limited bandwidth. The existing compressed video super-resolution methods pay more attention to the gain of low-frequency information in the video and process high-frequency information roughly. Besides, the geometric alignment information among temporal frames as well as the global information is also poorly extracted due to the limitation of the convolution operation. To address these limitations, we propose a Transformer based multi-level Alignments method to recover high-frequency and global information for compressed Video Super-Resolution (TAVSR). Specifically, a dual-branch alignment network is proposed. One branch is for recovering high-frequency information based on intra-frame which is compressed at original resolution; another branch is for low-frequency information in the continuous inter-frames at a lower resolution. For each branch, global and local alignments are performed respectively. To achieve global pixel movement alignment between the current frame and intra/inter-frame, Transformer based U-shape Network (TUNet) is proposed to estimate deformable convolution offsets, which performs much better than convolution in the geometric distance formulation from texture. By contrast, the local information is implicitly aligned using TUNet to keep the details. A multi-stage fusion module is further proposed to fuse aligned features to obtain the original resolution frame with enhanced quality. Extensive experiments show that the proposed method achieves the best rate-distortion (R-D) performance on JCT-VC test sequences compared with the most advanced methods.