Diagnose Before You Refine: Multi-Dimensional Judge Feedback for Query-Efficient Jailbreak Attacks
The paper introduces JSB, a query-efficient automated jailbreak method that uses multi-dimensional probabilistic judge feedback to diagnose why a prompt failed and then refine, switch strategies, or reseed from the best prior state within a small target-model query budget. What sets it apart is the tight coupling of fine-grained evaluator signals with bandit-style exploration and refinement, yielding strong reported success rates with only 10 queries, though it still depends on the quality and availability of the judge feedback and is evaluated mainly on standard harmful-request benchmarks rather than truly open-ended real-world attack settings.
Abstract
Automated jailbreak attacks rely on evaluation feedback to optimize adversarial prompts, but existing methods often lack fine-grained diagnostic information. We propose Jev-Guided Strategy Bandit (JSB), a query-based jailbreak framework that leverages multi-dimensional probabilistic feedback to guide prompt optimization. JSB combines continuous jailbreak scoring with targeted refinement, strategy switching, and best-state reseeding to balance exploration and refinement. We evaluate JSB on two harmful request datasets across five open-source and three closed-source LLMs. Under a budget of only 10 target-model queries, JSB achieves an average HarmBench attack success rate of 89.8\% across ten open-source settings, consistently outperforming existing automated jailbreak attacks. Further analyses demonstrate its effectiveness under varying query budgets and attacker models. Defense experiments also reveal limitations in existing filtering mechanisms and highlight the potential of Jev-based output filtering.