End-to-End Audiovisual Speech Recognition System With Multitask Learning

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

End-to-End Audiovisual Speech Recognition System With Multitask Learning

Fei Tao; Carlos Busso

An automatic speech recognition (ASR) system is a key component in current speech-based systems. However, the surrounding acoustic noise can severely degrade the performance of an ASR system. An appealing solution to address this problem is to augment conventional audio-based ASR systems with visual features describing lip activity. This paper proposes a novel end-to-end, multitask learning (MTL), audiovisual ASR (AV-ASR) system. A key novelty of the approach is the use of MTL, where the primary task is AV-ASR, and the secondary task is audiovisual voice activity detection (AV-VAD). We obtain a robust and accurate audiovisual system that generalizes across conditions. By detecting segments with speech activity, the AV-ASR performance improves as its connectionist temporal classification (CTC) loss function can leverage from the AV-VAD alignment information. Furthermore, the end-to-end system learns from the raw audiovisual inputs a discriminative high-level representation for both speech tasks, providing the flexibility to mine information directly from the data. The proposed architecture considers the temporal dynamics within and across modalities, providing an appealing and practical fusion scheme. We evaluate the proposed approach on a large audiovisual corpus (over 60 hours), which contains different channel and environmental conditions, comparing the results with competitive single task learning (STL) and MTL baselines. Although our main goal is to improve the performance of our ASR task, the experimental results show that the proposed approach can achieve the best performance across all conditions for both speech tasks. In addition to state-of-the-art performance in AV-ASR, the proposed solution can also provide valuable information about speech activity, solving two of the most important tasks in speech-based applications.

SPS on Twitter

  • DEADLINE EXTENDED: There's still time to submit your proposal to host the 2023 IEEE International Symposium on Biom… https://t.co/IyGtA3abJj
  • The 35th Picture Coding Symposium is heading to Bristol, UK and is now accepting papers for their June event! Head… https://t.co/sn8KNrKrBA
  • Deadline to submit to has been extended to 25 January! https://t.co/INpP2KrzoH
  • CALL FOR PAPERS: The 2021 IEEE International Conference on Autonomous Systems is now accepting papers for their Aug… https://t.co/IodWdsdLKc
  • The Brain Space Initiative Talk Series continues Friday, 15 January at 11:00 AM EST when Dr. Eva Dyer presents "Rep… https://t.co/fHP98guH0Z

SPS Videos

Signal Processing in Home Assistants


Multimedia Forensics

Careers in Signal Processing             


Under the Radar