Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents
The paper tests whether prompt-injection detector scores on public benchmarks actually predict performance when those detectors are used inside LLM agents to screen real tool outputs, using replayed AgentDojo and tau-bench tool calls plus BIPIA to compare fifteen detectors and task-aware LLM judges. Its key insight is that benchmark leaderboards can be deeply misleading because detection ability transfers poorly across settings while false-positive behavior does transfer, with training-input format explaining much of the gap; the main limitation is that the evaluation is still tied to two agent benchmarks and replayed ground-truth tool calls rather than fully live, messy deployments.
Abstract
LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmarks. We ask whether those scores predict how a detector behaves inside an agent. We replay the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM to obtain tool outputs that are benign by construction, label injected outputs by differential replay, and evaluate fifteen detectors, including Meta's Prompt Guard 2, and two task-aware LLM judges on these outputs and on the BIPIA benchmark. Detection rankings transfer poorly between benchmarks: the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector that catches 72% of AgentDojo injections catches 15% on tau-bench. False-positive rates on tool outputs, which range from none to over 90%, do transfer between the two agent benchmarks. Where training data is public, the form of the training inputs explains the results. The BIPIA leader was trained on full BIPIA inputs, but having seen InjecAgent's attack strings as short prompts does not help it find them inside tool outputs; the best detector on both agent benchmarks shares no data with any benchmark and was trained on agent-style inputs. Evaluations meant to inform deployment should use the agent's own tool outputs, report detection at a low false-positive rate, and audit what the detector was trained on.