TRACER: Trace-Regularized Training Against Indirect Prompt Injection
TRACER is a channel-aware adversarial training method for making LLM agents more robust to indirect prompt injection, showing that the same malicious instruction can be much more effective when delivered through retrieved context or tools than through normal chat turns. Its key idea is to train across realistic presentation channels while regularizing the model’s reasoning toward clean evidence, which yields big attack-success reductions on GSM8K/Qwen models without destroying clean accuracy, though the evidence is still limited to relatively small models, a narrow benchmark, and attack framings covered by the training/evaluation setup.
Abstract
Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior. We show that defense robustness is channel-conditioned: the same attack succeeds 18% of the time as a follow-up message but 53% when placed in retrieved context. We propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training method that samples presentation wrappers and combines an answer-anchored, evidence-gated reward with ATRC, which aligns attacked reasoning with clean evidence when an injection is invalid. On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1% while preserving 78-82% clean accuracy. Joint RAG+tool training achieves 13.9% average ASR across framings at 3B scale, while a single epoch at 7B reaches 9.4%. These results demonstrate that robust defenses must explicitly train and evaluate across deployment channels.