Multi-Memory Convolutional Neural Network for Video Super-Resolution

Video super-resolution (SR) is focused on reconstructing high-resolution frames from consecutive low-resolution (LR) frames. Most previous video SR methods based on convolutional neural networks (CNN) use a direct connection and single-memory module within the network, and thus, they fail to make full use of spatio-temporal complementary information from LR observed frames. To fully exploit spatio-temporal correlations between adjacent LR frames and reveal more realistic details, this paper proposes a multi-memory CNN (MMCNN) for video SR, cascading an optical flow network and an image-reconstruction network. A series of residual blocks engaged in utilizing intra-frame spatial correlations is proposed for feature extraction and reconstruction. Particularly, instead of using a single-memory module, we embed convolutional long short-term memory into the residual block, thus forming a multi-memory residual block to progressively extract and retain inter-frame temporal correlations between the consecutive LR frames. We conduct extensive experiments on numerous testing datasets with respect to different scaling factors. Our proposed MMCNN shows superiority over the state-of-the-art methods in terms of PSNR and visual quality and surpasses the best counterpart method by 1 dB at most.