COPA: Continual Preference Optimization for Adaptive Prompt Injection Defense
COPA frames prompt-injection defense for LLMs as a continual learning problem, updating the model with feedback from newly observed attacks via GRPO while replaying prior attack examples to avoid forgetting old defenses. The interesting bit is the margin-weighted experience replay for adaptive adversaries, which reportedly cuts attack success much more than static defenses, though it seems to rely on a steady stream of labeled/feedback-rich attacks and the abstract leaves open how costly, scalable, and broadly generalizable it is in messy real deployments.
Abstract
LLMs remain vulnerable to prompt injection attacks, where adversarial instructions embedded in user inputs or external content manipulate model behavior and bypass safeguards. Existing defenses are predominantly static, relying on fixed alignment objectives or attack-specific filtering mechanisms that require redesign as new attack strategies emerge. While recent lifelong alignment methods address shifting user preferences, they do not account for adaptive adversaries that continually evolve to exploit weaknesses in previously learned defenses. This limitation is particularly important in real-world deployments, where evolving attack distributions necessitate continual adaptation without sacrificing robustness to previously encountered threats. We present COPA, a continual preference optimization framework that treats prompt-injection defense as a lifelong learning problem. Instead of one-time alignment, COPA incrementally incorporates feedback from newly observed attacks via GRPO-based optimization and uses margin-weighted experience replay to retain defenses against prior attack classes. This enables continuous adaptation to emerging threats while mitigating catastrophic forgetting and preserving general-purpose model capabilities. Across lifelong prompt injection attack streams, COPA reduces attack success rate by up to 6.3x and 4.4x on average compared to state-of-the-art defenses. These results highlight continual preference optimization as an effective paradigm for defending LLMs against adaptive adversaries.