Advances and Challenges in Audio-Visual Sound Source Localization (video)

Audio-visual machine learning models are transforming video understanding by effectively integrating auditory and visual data, enabling diverse applications such as audio-visual scene analysis. A prominent example within this field is visual sound source localization, which identifies the spatial location of sounds in visual scenes, applicable to both environmental audio and speech. Recent methods leverage various intra and inter-modality model design choices, training-invariance strategies and data augmentations, leading to significant performance improvements. However, despite these advancements, current models often exhibit biases towards visual information, making it unclear how much each modality contributes to the final decision. Furthermore, existing task definitions typically prioritize visual perception, prevalent benchmarks can be noisy or unreliable, and standard evaluation metrics only partially capture the models' true performance. This webinar will discuss recent advancements in visual sound source localization, critically assess existing methodologies, present prevailing challenges, and highlight potential pathways for future research in addressing these issues.