When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs
The paper systematically evaluates LLM jailbreak defenses not just for safety, but for their side effects on downstream task performance, benign-input over-refusal, and inference cost across common defenses, benchmarks, and open-source models. Its useful twist is grouping defenses by operational strategy to show distinct trade-off patterns—rule-based methods tend to preserve utility, conservative self-reflection over-refuses, and multi-round defenses are expensive—though its conclusions are bounded by the particular benchmarks, defenses, and open-source models studied.
Abstract
Jailbreak defenses are essential for protecting large language models (LLMs), but they can also introduce secondary costs that weaken model utility. We present a systematic study of these defense trade-offs along three dimensions: performance impact, over-refusal on benign inputs, and inference cost. Rather than treating defenses as a single class, we organize them by operational strategy and examine how different strategies correlate with different side-effect profiles. Across state-of-the-art defense methods, widely used benchmark datasets, and representative open-source LLMs, we find that defenses rarely improve downstream capability, but instead vary in how they trade safety gains against usability and efficiency. In particular, rule-based defenses best preserve task performance, highly conservative self-reflective defenses often increase over-refusal, and multi-round defenses incur the largest runtime overhead. These results provide both a benchmark for evaluating defense side effects and practical guidance for selecting defenses under deployment constraints.