TVFormer: Trajectory-guided Visual Quality Assessment on 360° Images with Transformers

Visual quality assessment (VQA) on 360 degrees images plays an important role in optimizing immersive multimedia systems. Due to the absence of pristine 360 degrees images in real world, blind VQA (BVQA) on 360 degrees images has drawn much research attention. In subjective VQA on 360 degrees images, human intuitively make the quality-scoring decisions through the quality degradation of each observed viewport on the head trajectories. Unfortunately, the existing BVQA works for 360 degrees images neglect the dynamic property of head trajectories with viewport interactions, thus failing to obtain human-like quality scores. In this paper, we propose a novel Transformer-based approach for trajectory-guided VQA on 360 degrees images (named TVFormer), in which both the tasks of head trajectory prediction and BVQA can be accomplished for 360 degrees images. In the first task, we develop a trajectory-aware memory updater (TMU) module, for maintaining the coherence and accuracy of predicted head trajectories. To capture the long-range quality dependency across time-ordered viewports, we propose a spatio-temporal factorized self-attention (STF) module in the encoder of TVFormer for the BVQA task. By implanting the predicted head trajectories into the BVQA task, we can obtain the human-like quality scores. Extensive experiments demonstrate the superior BVQA performance of TVFormer over state-of-the-art approaches on three benchmark datasets.