Which Image Property Carries the Jailbreak? A Controlled Dissection of Image-to-Text Jailbreaks
The paper systematically tests which image-side properties actually drive image-to-text jailbreaks in multimodal models by holding the harmful text constant across four attack families, multiple VLMs, and controlled ablations over factors like visual density, tiling, and query-related image content. Its nice contribution is separating several previously conflated explanations: entropy/JPEG size and tile count don’t reliably explain success, while removing query-specific visual relatedness modestly reduces attack success in one setting, though the conclusions are still limited by the automated judge, incomplete measurement of image-text congruence, and relatively narrow statistically supported attribution.
Abstract
Image-to-text jailbreaks place harmful intent in text, image content, or the relationship between them. We examine image-side factors across four published attack families on a 313-prompt StrongREJECT slice, using five multimodal models and an additional appendix evaluation of InternVL3.5-8B. The harmful instruction is held constant across conditions; the baseline matrix uses one draw per prompt, and paired ablations use three draws with an automated rubric judge. A bare harmful query, with or without a benign unrelated image, produces little attack success on most victims, while attack images substantially increase it. First, per-tile entropy and JPEG size do not reliably distinguish attack tiles from size-matched benign distractors, limiting density-only screening. Second, earlier tile-count ladders were confounded by payload visibility. A corrected region-count test found no detectable effect, so the role of tile-count structure remains unresolved. Third, on Qwen3-VL-8B, the E4 manipulation that removes query-specific relatedness lowers ASR by about 0.12. This supports a bounded attribution to the relatedness manipulation, although image-text congruence remains unmeasured. A within-category control reproduces the direction on overlapping stimuli. Five tests survive the global statistical correction, but only E4 supports attribution to one measured descriptor; the within-category result is a robustness check, not a separate attribution. These conclusions remain conditional on the rubric judge.