Diffusion-based adversarial purification (AP) is an effective test-time defense that first adds noise to the input and then runs a reverse generative process to recover a clean sample. Its central challenge is to balance noise and semantics. The added noise must be strong enough to remove adversarial perturbations, but not so strong that it destroys the label-relevant content of the original sample. However, most existing methods add uniform noise, so weakly perturbed regions receive unnecessary noise, while heavily perturbed regions may be purified insufficiently. To address this challenge, we propose a pixel-adaptive noise schedule guided by a signal-to-noise ratio (SNR) map. This schedule assigns stronger noise to regions that are complex in content and easily exploited by adversarial attacks, and weaker noise to smooth background regions. To preserve semantic content, we further introduce an automatic stopping criterion that terminates the forward noising process before the injected noise overwhelms the signal. To obtain more reliable semantic latents and more accurate pixel-wise noise schedules, we design a two-stage latent optimization module that decouples the latent representations from adversarial inputs. Finally, extensive experimental results on the CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets demonstrate that our proposed method outperforms representative adversarial training and purification methods under most attacks, including a stronger adaptive white-box setting, while maintaining high standard accuracy.
