Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models
The paper argues that jailbreak evaluations based only on sampled attack success rate miss “nearby” harmful outputs in an LLM’s probability distribution, and proposes bounded continuation analysis to estimate the probability that autoregressive generation can reach a harmful continuation within a finite horizon. The nice idea is to treat generation as a probabilistic prefix tree and use model checking plus semantic harm labels to expose hidden risk even when the sampled response is a refusal, though the approach is inherently bounded, depends on the harm evaluator, and may get expensive or incomplete as the continuation space grows.
Abstract
LLM safety evaluations via jailbreaking typically index on attack success rate (ASR): whether a generated response violates safeguards. However, ASR only evaluates sampled outputs and provides little information on the underlying generation process. Autoregressive language models define distributions over possible outputs, and a sampled refusal can coexist with substantial probability mass on harmful generation paths. We study this hidden structure through bounded continuation analysis (BCA), which models autoregressive generation as a stochastic process over token prefixes and computes the finite-horizon probability that generation reaches a harmful continuation. BCA constructs a bounded continuation tree, labels terminal continuations with a semantic harm evaluator, and estimates harmful reachability using exact probabilistic model checking. BCA complements ASR by measuring how much harmful behavior remains accessible beneath a sampled safe response. This makes it useful for distinguishing models that look identical under measured ASR values, and allocating red-team effort toward regions of the generation process with greater hidden risk. Across four instruction-tuned LLMs and 100 JailbreakBench behaviors, we show that BCA effectively quantifies hidden harmful behavior and provides a useful signal to guide red teaming.