Action-Level Backdoor Attacks Against Deep Reinforcement Learning Systems via Adaptive Reward Exploration

Deep Reinforcement Learning (DRL) has demonstrated remarkable capabilities in domains such as robotics, finance, and autonomous systems. With the increasing cost of training, DRL models are increasingly shared and reused via model marketplaces, cloud platforms, and open-source repositories. This trend exposes DRL systems to action-level backdoor attacks, where adversaries inject malicious behaviors into models during the training phase and later manipulate their action outputs at deployment, potentially endangering safety-critical applications (e.g., causing vehicle collisions or drone crashes). Despite demonstrating feasibility, such attacks typically rely on handcrafted, task-specific backdoor rewards, which severely limit their generality and practicality, thereby causing practitioners to underestimate the true threat they pose. In this paper, we propose Adapdoor, the first general attack framework for adaptive backdoor reward exploration. Adapdoor coarsely initializes the backdoor reward from benign reward statistics and iteratively fine-tunes it based on the performance feedback from both benign and backdoor tasks. This enables stable action-level backdoor injection across diverse tasks without relying on extensive trial-and-error and expert knowledge. We conduct comprehensive experiments across 3 DRL algorithms, 11 environments, and 53 backdoor tasks, showing that Adapdoor outperforms existing baselines by 42.0% to 144.2%, under varying agent counts, action spaces, and reward signal types. Our findings reveal that action-level backdoors pose a more severe and realistic threat to DRL systems than previously recognized. Furthermore, we evaluate three potential defenses to explore pathways for mitigating this threat. The source code and task configurations of our evaluation are publicly available to ensure reproducibility and facilitate further research.