Tokens in Disarray: A Unified Adversarial Attack Against Multi-Layered Safeguards in LLMs and VLMs

Large Language Models (LLMs) and Vision-Language Models (VLMs) are typically equipped with multi-layered safety guardrails to prevent harmful content generation. Several attacks have been proposed to evaluate the robustness of these safeguarded models. However, current attacks rely on simplified or incomplete defense assumptions, limiting their effectiveness against models with multi-layered safeguards. We address this gap by formulating the problem of attacking generative models protected by comprehensive multi-layered defenses. To address this challenge, we propose a unified adversarial attack, the Token Disarray Attack (TDA), inspired by the differential handling of permuted tokens. Through internal inspection for harmful inputs with permuted tokens, we observe that while generative models retain comprehension of the input’s semantic meaning, their corresponding safety defenses are significantly compromised. Based on this observation, TDA first leverages LLMs to generate a diverse candidate dataset of token permutations for harmful inputs, and subsequently employs a genetic algorithm to optimize the harmfulness and bypass rate of permuted samples within this candidate dataset. Our evaluation demonstrates that TDA achieves an average attack success rate exceeding 70%, substantially outperforming state-of-the-art methods. When applied to commercial models, it achieves attack success rates of up to 53% on individual models, consistently outperforming existing methods and demonstrating notable cross-model transferability.