Expanding the Horizons: Exploring Further Steps in Open-Vocabulary Segmentation

The open vocabulary segmentation (OVS) task has gained significant attention due to the challenges posed by both segmentation and open vocabulary classification, which involves recognizing arbitrary categories. Recent studies have leveraged pretrained Vision-Language models (VLMs) as a new paradigm for addressing this problem, leading to notable achievements. However, our analysis reveals that these methods are not yet fully satisfactory. In this paper, we empirically analyze the key challenges in four main categories: segmentation, dataset, reasoning and recognition. Surprisingly, we observe that the current research focus in OVS primarily revolves around recognition issues, while others remain relatively unexplored. Motivated by these findings, we propose preliminary approaches to address the top three identified issues by integrating advanced models and making adjustments to existing segmentation models. Experimental results demonstrate the promising performance gains achieved by our proposed methods on the OVS benchmark.