Noise-Resilient Training Method for Face Landmark Generation From Speech

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

Noise-Resilient Training Method for Face Landmark Generation From Speech

By: 
Sefik Emre Eskimez; Ross K. Maddox; Chenliang Xu; Zhiyao Duan

Visual cues such as lip movements, when available, play an important role in speech communication. They are especially helpful for the hearing impaired population or in noisy environments. When not available, having a system to automatically generate talking faces in sync with input speech would enhance speech communication and enable many novel applications. In this article, we present a new system that can generate 3D talking face landmarks from speech in an online fashion. We employ a neural network that accepts the raw waveform as an input. The network contains convolutional layers with 1D kernels and outputs the active shape model (ASM) coefficients of face landmarks. To promote smoother transitions between video frames, we present a variant of the model that has the same architecture but also accepts the previous frame's ASM coefficients as an additional input. To cope with background noise, we propose a new training method to incorporate speech enhancement ideas at the feature level. Objective evaluations on landmark prediction show that the proposed system yields statistically significantly smaller errors than two state-of-the-art baseline methods on both a single-speaker dataset and a multi-speaker dataset. Experiments on noisy speech input with five types of non-stationary unseen noise show statistically significant improvements of the system performance thanks to the noise-resilient training method. Finally, subjective evaluations show that the generated talking faces have a significantly more convincing match with the input audio, achieving a similarly convincing level of realism as the ground-truth landmarks.

SPS on Twitter

  • THIS FRIDAY: Join our Vice President-Membership, K.V.S. Hari, and Membership Development Committee Chair, Arash Moh… https://t.co/rGSzhHAwgM
  • The SPACE webinar series continues tomorrow, Tuesday, 11 August at 11 AM ET with Dr. Xiao Xiang Zhu presenting "Dat… https://t.co/X5oz4KiJwX
  • now accepting submissions for special sessions, tutorials, and papers! The conference is set for June 2… https://t.co/sB3o5ItL0j
  • DEADLINE EXTENDED: The IEEE Journal of Selected Topics in Signal Processing is now accepting papers for a Special I… https://t.co/2SJwqj7aDB
  • NEW WEBINAR: Join us on Friday, 14 August at 11:00 AM ET for the 2021 SPS Membership Preview! Society leadership wi… https://t.co/1PLaZIt2VQ

SPS Videos


Signal Processing in Home Assistants

 


Multimedia Forensics


Careers in Signal Processing             

 


Under the Radar