Webinar: AudioLDM 2: Learning Holistic Audio Generation with Self-Supervised Pretraining (video)
"AudioLDM 2: Learning Holistic Audio Generation with Self-Supervised Pretraining" presented by Dr. Haohe Liu. Based on the IEEE Xplore® article under the same title. Published: IEEE Transactions on Audio, Speech, and Language Processing, May 2024. Visit article article here.
AudioLDM 2 is a holistic framework for unified audio generation that produces speech, music, and sound effects using a single model. Unlike prior approaches that require separate architectures with task-specific designs for each audio type, AudioLDM 2 introduces a general audio representation called the "language of audio" (LOA), learned through AudioMAE, a self-supervised pretrained model. During generation, a GPT-2 model translates input conditions such as text into LOA, which then guides a latent diffusion model to synthesize high-quality audio. This design unifies diverse audio generation tasks under one framework while enabling advantages such as in-context learning and reusable pretrained components.
The talk will cover the motivation behind holistic audio generation, the AudioLDM 2 architecture, key experimental findings, practical lessons learned from building a unified audio generation system, and recent advancement in this rapidly evolving field.
