Geometric Configurations of Perturbed Jailbreak Prompts
The paper analyzes how string-level perturbations that can turn failed jailbreak prompts into successful ones are represented inside small Qwen and Llama instruct models, comparing last-token hidden embeddings with top-50 next-token probability vectors. Its useful angle is showing that hidden embeddings mostly reflect surface form while next-token distributions collapse to a near-1D structure and offer little evidence of a clean “jailbreak behavior” boundary, though the study is limited to small models, a refusal-heavy dataset, and only a few representation spaces with weak token-level signals.
Abstract
Perturbation techniques that turn unsuccessful jailbreak prompts into successful ones are continuously evolving, constituting a major security threat to LLM safety. In this paper, we investigate the internal representations of such string-level perturbed jailbreak inputs in the small weight models of the Qwen-2.5-1.5B/-3B/-7B-Instruct and Llama-3.2-1B/-3B/-3.1-8B-Instruct families. We select two representation spaces: the last-layer-last-token embedding space and the top-50 next-token probability space. The former space separates prompts based on their spelling and format, while the latter space is effectively one-dimensional but appears more complex to cluster. Within our refusal-dominated answer set we find no behavioral hyperplane in either space. Only the next token "Sure" in the 1.5B Qwen model, and both tokens "," and "ĊĊ" in the 1$ Llama model, display a significant association with a compliant-labeled answer.