ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents
ToolHazard is a framework for automatically synthesizing executable, stateful tool-use environments with adversarial prompt-injection opportunities, then using them to benchmark and train LLM agents on long-horizon tasks under realistic indirect attacks. Its neat contribution is scaling beyond hand-built benchmarks by having simulator/attacker/user components discover injection points, craft context-specific payloads, and generate alignment data that transfers to another benchmark, though the evidence still hinges on synthesized environments and the abstract doesn’t establish how well the setup matches messy real-world tool ecosystems.
Abstract
Large language model (LLM) agents integrated with external tools are vulnerable to indirect prompt injections embedded in environmental states. However, existing studies largely rely on manually implemented or reused environments, stochastic LLM-based tool simulation, and predefined injection locations, limiting scalable security research across broader domains. To bridge this gap, we propose **ToolHazard**, a scalable adversarial environment synthesis framework that reduces human engineering and supports expansion with additional seed domains and compute. Through an Environment Simulator, an Attacker Agent, and a User Simulator, ToolHazard synthesizes executable stateful environments, discovers viable injection points and generates environment-specific payloads, and constructs state-grounded long-horizon tasks. Based on ToolHazard, we build **ToolHazard-Bench** for stress-testing agents under complex workflows and diverse environmental attacks. Experiments reveal substantial agent vulnerabilities and show that injection timing and placement affect attack effectiveness. Moreover, ToolHazard-generated alignment data improves security on both ToolHazard-Bench and AgentDojo while preserving benign task utility.