MRRVOS: Modular Refinement Referring Video Object Segmentation

The previous referring video object segmentation method only focuses on the final prediction result. When the user input linguistic query does not match the information in the video, the video still segments error information. We propose a new reference video object segmentation framework. In our model, which we call the Modular Refinement Referring Video Object Segmentation (MRRVOS), when the objects in the referring linguistic query do not match the video frame, stop segmentation and feedback error information. Firstly, given a video clip and a linguistic query. Our method segments the specified object in a video frame automatically. Then our method matches linguistic query with video frames and implements object segmentation on other frames through a recursive three-module model: (1) Referring Video Object Segmentation module: we consider the referring video object segmentation task as a joint problem of referring object segmentation in the image and mask propagation in the video. (2) Image caption module: using recurrent neural networks (RNNs), and deep convolutional neural network (CNN) to encode every frame except the first frame, and a Long Short Term Memory (LSTM) RNN decoder to generate the output caption and put it into the corpus. (3) Semantic dissimilarity module: put all the text results into the corpus and embed vector space, our linguistic query of the input to perform a semantic dissimilarity search. We show that our approach is competitive to the state-of-the-art method.