Explicit Visual Prompting for Low-Level Structure Segmentations

被引:51
作者
Liu, Weihuang [1 ]
Shen, Xi [2 ]
Pun, Chi-Man [1 ]
Cun, Xiaodong [2 ]
机构
[1] Univ Macau, Taipa, Macao, Peoples R China
[2] Tencent AI Lab, Shenzhen, Peoples R China
来源
2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR) | 2023年
关键词
D O I
10.1109/CVPR52729.2023.01862
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We consider the generic problem of detecting low-level structures in images, which includes segmenting the manipulated parts, identifying out-of-focus pixels, separating shadow regions, and detecting concealed objects. Whereas each such topic has been typically addressed with a domain-specific solution, we show that a unified approach performs well across all of them. We take inspiration from the widely-used pre-training and then prompt tuning protocols in NLP and propose a new visual prompting model, named Explicit Visual Prompting (EVP). Different from the previous visual prompting which is typically a dataset-level implicit embedding, our key insight is to enforce the tunable parameters focusing on the explicit visual content from each individual image, i.e., the features from frozen patch embeddings and the input's high-frequency components. The proposed EVP significantly outperforms other parameter-efficient tuning protocols under the same amount of tunable parameters (5.7% extra trainable parameters of each task). EVP also achieves state-of-the-art performances on diverse low-level structure segmentation tasks compared to task-specific solutions. Our code is available at: https://github.com/NiFangBaAGe/Explicit-Visual-Prompt.
引用
收藏
页码:19434 / 19445
页数:12
相关论文
共 89 条
  • [1] [Anonymous], 2009, IMAGE VISION COMPUTI
  • [2] Bae Soonmin, 2007, COMPUTER GRAPHICS FO, V1, P2
  • [3] Bar A., 2022, Advances in Neural Information Processing Systems
  • [4] Brown T., 2020, ADV NEURAL INF PROCE, V33, P1877
  • [5] Chen Shoufa, 2022, Advances in Neural Information Processing Systems
  • [6] Chen Zhihao, 2020, P C COMP VIS PATT RE
  • [7] Cucchiara Rita, 2003, T PATTERN ANAL MACHI, P3
  • [8] Cun X., 2018, P EUR C COMP VIS WOR, P1
  • [9] Cun Xiaodong, 2020, P EUR C COMP VIS ECC
  • [10] Cun Xiaodong, 2020, P AAAI C ART INT AAA