Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models
CNRF is a selective safety framework for vision-language models that builds benign–harmful counterfactual pairs, treats their representation differences as evidence-backed risk directions, separately estimates whether a query is in a supported neighborhood, and then intervenes or reviews only when the risk estimate is applicable. Its key idea is separating “looks unfamiliar” from “looks harmful,” which makes the defense more calibrated and less blanket-rejection-like than standard representation classifiers, though the abstract leaves open how well it scales beyond the tested VLMs/jailbreak families and how costly or brittle the counterfactual construction and victim-specific review pipeline are.
Abstract
Representation-based safety defenses for vision-language models (VLMs) often learn to separate harmful from benign inputs, but separation alone does not establish what a risk score means or when it should be trusted. We introduce Counterfactual Neighborhood Risk Fields (CNRF), an evidence-grounded framework for selective VLM safety. CNRF constructs context-matched benign-harmful counterfactual pairs, uses their representation differences as capability-related risk directions, and estimates neighborhood support separately from risk. This separation prevents unfamiliarity from being conflated with harmfulness and enables selective intervention rather than blanket rejection. A victim-adaptive interface accommodates different safety observations across VLM architectures and routes selected requests for same-victim response review, which either releases acceptable candidates or produces constrained alternatives with reduced harmful capability. Across three VLMs and five multimodal jailbreak families, CNRF achieves 86.06-89.30% macro defense success rate while retaining substantial benign-task utility. Controlled experiments further show that query-matched references and preserved context-direction correspondence materially improve risk ranking in the low-false-positive regime. CNRF thus reframes representation-based VLM safety from learning harmfulness boundaries to constructing applicable evidence for selective risk decisions.