Exploiting recollection effects for memory-based video object segmentation

Recent advances in deep learning have led to numerous studies on video object segmentation (VOS). Memory-based models, in particular, have demonstrated superior performance by leveraging the ability to store and recall information from previous frames. While extensive research efforts have been devoted to developing memory networks for effective VOS, only a few studies have investigated the quality of memory in terms of determining which information should be stored. In fact, in most recent memory-based VOS studies, the frame information is regularly stored in the memory without specific consideration. In other words, there is a lack of explicit criteria or guidelines for determining the essential information that should be retained in memory. In this study, we introduce a new method for evaluating the effect of storing the features, which can be used for various memory-based networks to improve performance in a plug-and-play manner. For this purpose, we introduce the concept of recollection effects, which refers to the stability of predictions based on the presence or absence of specific features in memory. By explicitly measuring the recollection effects, we establish a criterion for evaluating the relevance of information and determining whether features from a particular frame should be stored. This approach effectively encourages memory-based networks to construct memory that contains valuable cues. To validate the effectiveness of our method, we conduct comparative experiments. Experimental results demonstrate the effectiveness of our method to enhance the selection and retention of useful cues within the memory, leading to improving segmentation results. © 2023 Elsevier B.V.