An Empirical Measurement of Jailbreaking Evaluators
The paper benchmarks six commonly used automated jailbreak evaluators against human labels on the same datasets, controlling LLM-judge variation with a shared backbone where applicable, to see how reliably they measure jailbreak success. Its useful contribution is making evaluator choice itself the object of measurement—showing that JADES performs best overall while HarmBench and StrongReject are also strong—but its conclusions are still bounded by the two human-labeled datasets, the chosen backbone, and the particular attack families covered.
Abstract
Expert evaluation of jailbreak responses is costly and difficult to scale, so the community increasingly relies on automated evaluators to determine whether an attack succeeds. However, jailbreak studies typically validate their chosen evaluator independently, repeatedly spending resources on similar evaluation efforts while making results across papers difficult to compare. Different evaluators also encode different definitions of jailbreak success, meaning that reported attack strength and apparent progress can depend substantially on which evaluator is used. We systematically compare six evaluators that recur in recent jailbreak attack and defense research: HarmBench, JailbreakBench, JailbreakRadar, StrongReject, JADES, and JailMeter. To our knowledge, no prior study has evaluated all six on the same human-labeled data under a controlled setup. We evaluate them on JailbreakQR and JailMeter-Eva, using human judgments as the reference, and measure agreement with humans, error types, and consistency across attack families. For evaluators that require a general-purpose LLM judge, we use a shared backbone to control for model-specific variation. We found that JADES exhibits the best overall performance, while HarmBench and StrongReject also demonstrate good performance.