Audio-visual localization based on spatial relative sound order

被引:0
作者
Sato, Tomoya [1 ]
Sugano, Yusuke [1 ]
Sato, Yoichi [1 ]
机构
[1] Univ Tokyo, Inst Ind Sci, Tokyo, Japan
关键词
Computer vision; Machine learning; Audio-visual learning; Sound localization;
D O I
10.1007/s00138-025-01700-0
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Sound localization is one of the essential tasks in audio-visual learning. Especially, stereo sound localization methods have been proposed to handle multiple sound sources. However, existing stereo-sound localization methods treat sound source localization as a segmentation task and, as a result, require costly annotation of segmentation masks. Another serious problem of the existing stereo-sound localization methods is that they have been trained and evaluated only in a controlled environment, such as a fixed camera and microphone setting with limited variability of scenes. Therefore, their performance on videos recorded in uncontrolled environments, such as in-the-wild videos from the Internet, has not been fully investigated. To address these problems, we propose a weakly supervised method as an extension of a typical stereo-sound localization method by utilizing the spatial relative order of sound sources in recorded videos. The proposed method solves the annotation problem by training the localization model using only sound category labels. Furthermore, our method utilizes the spatial relative order of the sound sources, which is not affected by specific recording settings, and thus can be effectively used for videos recorded in uncontrolled environments. We also collect stereo-recorded videos from YouTube to construct a new dataset to demonstrate the applicability of the proposed method to stereo sounds recorded in various environments. Our method enhances the localization performance by inserting a novel training step that exploits the relative order of sound sources into a typical audio-visual localization method in both existing and newly introduced audio-visual datasets.
引用
收藏
页数:14
相关论文
共 50 条
[41]   A Novel Audio-Visual Information Fusion System for Mental Disorders Detection [J].
Li, Yichun ;
Li, Shuanglin ;
Naqvi, Syed Mohsen .
2024 27TH INTERNATIONAL CONFERENCE ON INFORMATION FUSION, FUSION 2024, 2024,
[42]   Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics [J].
Liu, Chen ;
Li, Peike Patrick ;
Qi, Xingqun ;
Zhang, Hu ;
Li, Lincheng ;
Wang, Dadong ;
Yu, Xin .
PROCEEDINGS OF THE 31ST ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA, MM 2023, 2023, :7590-7598
[43]   Audio-Visual Generalized Zero-Shot Learning the Easy Way [J].
Mo, Shentong ;
Morgado, Pedro .
COMPUTER VISION - ECCV 2024, PT LXXI, 2025, 15129 :377-395
[44]   Spatial Perception Under Visual Restriction by Moving a Sound Source Using 3D Audio [J].
Murakami, Koki ;
Watanuki, Keiichi ;
Kaede, Kazunori .
ADVANCES IN INDUSTRIAL DESIGN, AHFE 2021, 2021, 260 :757-764
[45]   STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking [J].
Li, Yidi ;
Liu, Hong ;
Yang, Bing .
IEEE TRANSACTIONS ON MULTIMEDIA, 2025, 27 :1835-1847
[46]   Contrastive Positive Sample Propagation Along the Audio-Visual Event Line [J].
Zhou, Jinxing ;
Guo, Dan ;
Wang, Meng .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2023, 45 (06) :7239-7257
[47]   CPM: Class-Conditional Prompting Machine for Audio-Visual Segmentation [J].
Chen, Yuanhong ;
Wang, Chong ;
Liu, Yuyuan ;
Wang, Hu ;
Carneiro, Gustavo .
COMPUTER VISION - ECCV 2024, PT X, 2025, 15068 :438-456
[48]   Sound propagation model for sound source localization in area of observation of an audio robot [J].
Pleshkova, Snejana ;
Bekiarski, Alexander .
PROCEEDINGS OF THE 9TH WSEAS INTERNATIONAL CONFERENCE ON NEURAL NETWORKS (NN' 08): ADVANCED TOPICS ON NEURAL NETWORKS, 2008, :119-122
[49]   Evolutionary structure of hidden Markov models for audio-visual Arabic speech recognition [J].
Makhlouf, Amina ;
Lazli, Lilia ;
Bensaker, Bachir .
INTERNATIONAL JOURNAL OF SIGNAL AND IMAGING SYSTEMS ENGINEERING, 2016, 9 (01) :55-66
[50]   Text-to-Feature Diffusion for Audio-Visual Few-Shot Learning [J].
Mercea, Otniel-Bogdan ;
Hummel, Thomas ;
Koepke, A. Sophia ;
Akata, Zeynep .
PATTERN RECOGNITION, DAGM GCPR 2023, 2024, 14264 :491-507