Training-Free Diffusion-Based Conditional Generation With Collaborative Samples

Recent advances in unconditional diffusion models have shown remarkable performance in conditional generation tasks. Typically, diffusion models serve as priors, incorporating differentiable loss functions in these tasks without requiring retraining. Traditionally, outputs are mutually independent during the sampling process, which limits the exploration of the solution space and neglects the underlying structure of the conditional distribution at the sample level. To address this challenge, we propose S-diff, a novel plug-and-play framework built upon Stein variational gradient descent (SVGD). This framework enhances performance in conditional generation tasks by exploring the interactions among samples during the diffusion sampling process. Specifically, we treat multiple samples as “particles” in the SVGD update and combine the score function of diffusion models at the first timestep with an off-the-shelf differentiable loss function to approximate the conditional score function. This conditional score is subsequently smoothed through kernels defined by other particles, collectively guiding particles toward regions of higher probability. Importantly, approximation errors remain well-bounded owing to the intrinsic properties of the diffusion model. This framework can be seamlessly integrated into both the common consecutive diffusion sampling process and the decoupled sampling process. Our experiments demonstrate the effectiveness of S-diff across a wide range of conditional generation tasks. For example, we comprehensively evaluate S-diff across 16 conditional generation tasks on a challenging benchmark, covering vision, audio, and molecular domains. Results show that S-diff achieves an average relative improvement of 9.4%.