A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural F0 Model for Statistical Parametric Speech Synthesis

You are here

Top Reasons to Join SPS Today!

1. IEEE Signal Processing Magazine
2. Signal Processing Digital Library*
3. Inside Signal Processing Newsletter
4. SPS Resource Center
5. Career advancement & recognition
6. Discounts on conferences and publications
7. Professional networking
8. Communities for students, young professionals, and women
9. Volunteer opportunities
10. Coming soon! PDH/CEU credits
Click here to learn more.

A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural F0 Model for Statistical Parametric Speech Synthesis

Xin Wang; Shinji Takaki; Junichi Yamagishi; Simon King; Keiichi Tokuda

Recurrent neural networks (RNNs) can predict fundamental frequency (F 0 ) for statistical parametric speech synthesis systems, given linguistic features as input. However, these models assume conditional independence between consecutive F 0 values, given the RNN state. In a previous study, we proposed autoregressive (AR) neural F 0 models to capture the causal dependency of successive F 0 values. In subjective evaluations, a deep AR model (DAR) outperformed an RNN. Here, we propose a Vector Quantized Variational Autoencoder (VQ-VAE) neural F 0 model that is both more efficient and more interpretable than the DAR. This model has two stages: one uses the VQ-VAE framework to learn a latent code for the F 0 contour of each linguistic unit, and other learns to map from linguistic features to latent codes. In contrast to the DAR and RNN, which process the input linguistic features frame-by-frame, the new model converts one linguistic feature vector into one latent code for each linguistic unit. The new model achieves better objective scores than the DAR, has a smaller memory footprint and is computationally faster. Visualization of the latent codes for phones and moras reveals that each latent code represents an F 0 shape for a linguistic unit.

SPS on Twitter

  • IEEE Day is October 4th. Celebrate IEEE Day by attending a local event. Visit the IEEE Day site for a complete list… https://t.co/mESJHTn7ek
  • The Biomedical Imaging and Signal Processing Webinar Series continues on Tuesday, 4 October when Selin Aviyente pre… https://t.co/Gl4bHlWbqh
  • On Wednesday, 26 October, join Dr. DeLiang Wang for a new SPS webinar, "Neural Spectrospatial Filter" - register no… https://t.co/vUkiWC4Am8
  • Join Dr. Peilan Wang and Dr Jun Fang for "Channel State Information Acquisition for Intelligent Reflecting Surface-… https://t.co/jOhyA10xuG
  • The SPS Webinar Series continues on Monday, 10 October when Dr. Luisa Verdoliva presents "Media Forensics and DeepF… https://t.co/aInDvTSQZc

SPS Videos

Signal Processing in Home Assistants


Multimedia Forensics

Careers in Signal Processing             


Under the Radar