Pyramid-structured multi-scale transformer for efficient semi-supervised video object segmentation with adaptive fusion

In recent years, Transformer-based methods have demonstrated promising performance in the field of semi-supervised video object segmentation. However, these methods require the maintenance of a memory frame from memory bank, which leads to an exponential increase in GPU memory requirements as the length of the video increases, necessitating updates of the memory bank every few frames. We propose a novel approach based on a multi-scale pyramid structure for object association with transformers, which can effectively encode both global and local features at different granularity levels, while significantly reducing GPU memory requirements as video length increases, thus maintaining high inference speed. To effectively integrate multi-scale ID embeddings and video frame embeddings, rather than simply overlaying the original features through addition, we have designed an adaptive fusion module to address this issue. We conducted extensive experiments on four commonly used VOS benchmarks (including YouTube-VOS 2018 and 2019 Val, DAVIS-2017,and LVOS), evaluating various variants of AOT. Our method outperformed state-of-the-art competitors and consistently demonstrated superior efficiency and scalability across all four benchmark tests.