Audio-Visual Segmentation with Semantics

被引:0
作者
Zhou, Jinxing [1 ]
Shen, Xuyang [2 ]
Wang, Jianyuan [3 ]
Zhang, Jiayi [4 ]
Sun, Weixuan [2 ]
Zhang, Jing [5 ]
Birchfield, Stan [6 ]
Guo, Dan [1 ]
Kong, Lingpeng [7 ]
Wang, Meng [1 ]
Zhong, Yiran [2 ]
机构
[1] Hefei Univ Technol, Hefei, Peoples R China
[2] Shanghai AI Lab, Shanghai, Peoples R China
[3] Univ Oxford, Oxford, England
[4] Beihang Univ, Beijing, Peoples R China
[5] Australian Natl Univ, Canberra, Australia
[6] Nvidia, Santa Clara, CA USA
[7] Univ Hong Kong, Hong Kong, Peoples R China
关键词
Audio-visual segmentation; Multi-modal segmentation; Audio-visual learning; AVSBench; Semantic segmentation; Video segmentation; SOUND;
D O I
10.1007/s11263-024-02261-x
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We propose a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark, i.e., AVSBench, providing pixel-wise annotations for sounding objects in audible videos. It contains three subsets: AVSBench-object (Single-source subset, Multi-sources subset) and AVSBench-semantic (Semantic-labels subset). Accordingly, three settings are studied: 1) semi-supervised audio-visual segmentation with a single sound source; 2) fully-supervised audio-visual segmentation with multiple sound sources, and 3) fully-supervised audio-visual semantic segmentation. The first two settings need to generate binary masks of sounding objects indicating pixels corresponding to the audio, while the third setting further requires to generate semantic maps indicating the object category. To deal with these problems, we propose a new baseline method that uses a temporal pixel-wise audio-visual interaction module to inject audio semantics as guidance for the visual segmentation process. We also design a regularization loss to encourage audio-visual mapping during training. Quantitative and qualitative experiments on the AVSBench dataset compare our approach to several existing methods for related tasks, demonstrating that the proposed method is promising for building a bridge between the audio and pixel-wise visual semantics. Code can be found at https://github.com/OpenNLPLab/AVSBench.
引用
收藏
页码:1644 / 1664
页数:21
相关论文
共 85 条
  • [1] Afouras Triantafyllos, 2020, COMPUTER VISIONECCV
  • [2] Objects that Sound
    Arandjelovic, Relja
    Zisserman, Andrew
    [J]. COMPUTER VISION - ECCV 2018, PT I, 2018, 11205 : 451 - 466
  • [3] Look, Listen and Learn
    Arandjelovic, Relja
    Zisserman, Andrew
    [J]. 2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 2017, : 609 - 617
  • [4] Aytar Y, 2016, ADV NEUR IN, V29
  • [5] End-to-End Referring Video Object Segmentation with Multimodal Transformers
    Botach, Adam
    Zheltonozhskii, Evgenii
    Baskin, Chaim
    [J]. 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), 2022, : 4975 - 4985
  • [6] One-Shot Video Object Segmentation
    Caelles, S.
    Maninis, K. -K.
    Pont-Tuset, J.
    Leal-Taixe, L.
    Cremers, D.
    Van Gool, L.
    [J]. 30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, : 5320 - 5329
  • [7] Chen H., 2021, BRIT MACH VIS C BMVC, P1
  • [8] Localizing Visual Sounds the Hard Way
    Chen, Honglie
    Xie, Weidi
    Afouras, Triantafyllos
    Nagrani, Arsha
    Vedaldi, Andrea
    Zisserman, Andrew
    [J]. 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021, 2021, : 16862 - 16871
  • [9] Chen HL, 2020, INT CONF ACOUST SPEE, P721, DOI [10.1109/icassp40776.2020.9053174, 10.1109/ICASSP40776.2020.9053174]
  • [10] DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
    Chen, Liang-Chieh
    Papandreou, George
    Kokkinos, Iasonas
    Murphy, Kevin
    Yuille, Alan L.
    [J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2018, 40 (04) : 834 - 848