AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
AgentSafeEvo is a reinforcement-learning framework that trains LLM agents for safer tool use via an attacker–defender game, where an attacker generates adversarial multi-turn agent tasks and the defender learns to complete them without being manipulated by malicious user inputs, tool descriptions, or observations. Its main twist is treating agent safety as co-evolution over diverse task/attack types plus a learnability-guided weighting scheme based on defender reward variability, which is more agent-specific than prior prompt-injection or language-only safety work, though its strength likely depends on how well the generated attack/task distribution and reward signals cover real-world failures.
Abstract
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.