The Fragility of Jailbreak Robustness Across Operational States
The paper shows that LLM jailbreak robustness is not a fixed property of a model-attack pair: across seven aligned models and three jailbreak methods, merely changing benign-looking system prompts can swing attack success rates dramatically, even when the jailbreak itself is unchanged. Its key insight is that these operational-state effects correlate with shifts in hidden activations along a refusal-related representation axis that predicts jailbreak outcomes, though the work seems mainly diagnostic and limited to a small set of models, attacks, and prompt-induced states rather than offering a complete robustness fix.
Abstract
Existing jailbreak evaluations typically characterize robustness using a single attack success rate (ASR) measured in a default configuration (the vanilla state). However, user-LLM interactions can induce diverse operational states beyond the vanilla state. In this work, we find that jailbreak robustness is highly fragile to operational-state variation: even when the attack remains fixed, changing only an ordinary system prompt not designed to affect safety can dramatically alter attack success rates. We systematically investigate this phenomenon across seven aligned models and three representative jailbreak attacks, observing substantial differences in ASR between vanilla and non-vanilla operational states. In one case, ASR increases by up to 56 percentage points (2% to 58%) solely due to a change in operational state. Remarkably, these increases occur even for attacks originally designed and optimized under vanilla-state evaluation. We further show that state-dependent robustness variation is systematically associated with differences in hidden representations along a refusal-related axis, and that projections onto this axis strongly predict jailbreak outcomes. Our results show that a single vanilla-state evaluation may not fully characterize jailbreak robustness, motivating evaluations that also examine how robustness changes across non-vanilla operational states.