Sampling Diffusion Models for Real-World Inverse Problems
In this SPS Distinguished Industry Lecturer highlight, NVIDIA’s Morteza Mardani shares a practical, task-agnostic way to use diffusion models for image restoration and reconstruction—without retraining foundation models.
Generative AI is changing how we process and reconstruct visual data across sectors—from inspection and media to biomedical imaging. In this talk, Morteza Mardani (Senior Research Scientist, NVIDIA; Visiting Researcher, Stanford EE) shows how to treat diffusion sampling as optimization, using score evaluation for controllable, plug-and-play guidance.
The approach connects Bayesian intuition with score-based modeling to solve inverse problems—including inpainting, super-resolution, and MRI reconstruction—often improving fidelity while avoiding fine-tuning of large diffusion backbones. Mardani also links the method to regularization-by-denoising and score-distillation ideas used in 2D→3D pipelines.
Key Highlights
- Plug-and-play diffusion: Use score guidance to solve downstream tasks without retraining the base model.
- Optimization view: Cast posterior sampling as stochastic optimization for interpretability and control (learning-rate/step control, task-specific constraints).
- Inverse problems in practice: Sharper reconstructions demonstrated on inpainting, super-resolution, and MRI—with attention to hallucination risk and deployment trade-offs.
- Foundation-model friendly: Works with existing diffusion priors; complements NeRF/3D pipelines via score-distillation connections.
- Industry takeaways: Easier deployment and tuning than many GAN-based routes for constrained vision tasks.
Distinguished Industry Lecturer
Morteza Mardani is a Senior Research Scientist at NVIDIA focusing on generative learning and a visiting researcher in Stanford University’s Electrical Engineering department. He previously held research roles at Stanford and the RISE Lab at UC Berkeley. He earned his Ph.D. in Electrical Engineering from the University of Minnesota (2015) and received the IEEE SPS Young Author Best Paper Award (2017).
