Real-Time Audio-Guided Multi-Face Reenactment

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

Real-Time Audio-Guided Multi-Face Reenactment

By: 
Jiangning Zhang; Xianfang Zeng; Chao Xu; Yong Liu

Audio-guided face reenactment aims to generate authentic target faces that have matched facial expression of the input audio, and many learning-based methods have successfully achieved this. However, most methods can only reenact a particular person once trained or suffer from the low-quality generation of the target images. Also, nearly none of the current reenactment works consider the model size and running speed that are important for practical use. To solve the above challenges, we propose an efficient A udio-guided M ulti-face reenactment model named AMNet , which can reenact target faces among multiple persons with corresponding source faces and drive signals as inputs. Concretely, we design a Geometric Controller (GC) module to inject the drive signals so that the model can be optimized in an end-to-end manner and generate more authentic images. Also, we adopt a lightweight network for our face reenactor so that the model can run in real-time on both CPU and GPU devices. Abundant experiments prove our approach’s superiority over existing methods, e.g ., averagely decreasing FID by 0.12  and increasing SSIM by 0.031  than APB2Face, while owning fewer parameters ( ×4 ) and faster CPU speed ( ×4 ).

Audio-guided face reenactment aims to generate authentic target faces under the condition of audio information along with auxiliary pose and eye blink signals, which has promising applications such as animation production, virtual human, and game. However, most current methods can only reenact a particular person once finishing the training procedure or suffer from the low-quality problem of the generated target images. Also, nearly none of the current reenactment works take the model size and running speed into account that is important for practical use. This work focuses on solving the above problems, and we improve previous APB2Face [1] to an efficient end-to-end model to handle audio-guided multi-face reenactment, where different target faces among multiple persons can be reenacted by only one unified model.

SPS on Twitter

  • Join Dr. Peilan Wang and Dr Jun Fang for "Channel State Information Acquisition for Intelligent Reflecting Surface-… https://t.co/jOhyA10xuG
  • The SPS Webinar Series continues on Monday, 10 October when Dr. Luisa Verdoliva presents "Media Forensics and DeepF… https://t.co/aInDvTSQZc
  • DEADLINE EXTENDED: The IEEE Transactions on Multimedia is accepting submissions for a Special Issue on Point Cloud… https://t.co/UqoOXUd8BH
  • Short courses return to ! Register for live and remote sessions, "A Hands-on Approach for Implementing Sto… https://t.co/qMoR6iqp4F
  • Join Dr. Sabyasachi Ghosh on Wednesday, 21 September for a new SPS Webinar, “Tapestry: A Compressed Sensing Approac… https://t.co/MNhu1kBmxG

SPS Videos


Signal Processing in Home Assistants

 


Multimedia Forensics


Careers in Signal Processing             

 


Under the Radar