Saliency Detection for Semantic Segmentation of Videos

There has been remarkable progress in the field of Semantic segmentation in recent years. Yet, it remains a challenging problem to apply segmentation to the video-based applications. Videos usually involve significantly larger volume of data compared to images. Particularly, a video contains around 30 frames per second. Segmentation of the similar frames unnecessarily adds to the time required for segmentation of complete video. In this paper, we propose a contour detection-based approach for detection of salient frames for faster semantic segmentation of videos. We propose to detect the salient frames of the video and pass only the salient frames through the segmentation block. Then, the segmented labels of the salient frames are mapped to the non-salient frames. The salient frame is defined by the variation in the pixel values of the background subtracted frames. The background subtraction is done using MOG2 background subtractor algorithm for background subtraction in various lighting conditions. We demonstrate the results using the Pytorch model for semantic segmentation of images. We propose to concatenate the semantic segmentation model to our proposed framework. We evaluate our result by comparing the time taken and the mean Intersection over Union (mIoU) for segmentation of the video with and without passing the video input through our proposed framework. We evaluate the results of Saliency Detection Block using Retention and Condensation ratio as the quality metrics.