MobileJailBench: Red Queen Jailbreak Benchmarking for Mobile Agents
MobileJailBench is a jailbreak benchmark for MLLM-based mobile agents that evaluates attacks in realistic app workflows using a mobile sandbox, 520 tasks across three jailbreak styles and 14 risk categories, and metrics that look beyond simple refusal or attack success. Its main contribution is treating mobile-agent safety as an evolving red-team problem—mutating jailbreaks from failures and expanding the benchmark around discovered weaknesses—which exposes that current agents can be superficially aligned in chat yet highly vulnerable during dynamic phone interactions, though its conclusions still depend on the fidelity of the sandbox, task coverage, and adjudication design.
Abstract
MLLM-based mobile agents are rapidly gaining autonomy in perception, planning, and closed-loop execution in dynamic mobile environments. However, this autonomy also broadens the attack surface and amplifies the risks of malicious misuse, including privacy leakage, device compromise, and large-scale phishing. Existing jailbreak evaluations leave three critical gaps for mobile agents: they focus on conversational attacks instead of mobile workflows, use static task sets with fixed difficulty, and rely narrowly on attack success rate rather than comprehensive security assessment. To close these gaps, we present MobileJailBench, the first evolving jailbreak benchmark for mobile agents, comprising three key components: (1) a benchmark comprising 520 tasks spanning three jailbreak paradigms and 14 risk categories, built on a high-fidelity mobile sandbox that we construct, enabling realistic workflows across over 20 widely used apps; (2) Dual-Dynamic Evolution, which combines Failure-Distilled Jailbreak Evolution with Weakness-Driven Benchmark Expansion; and (3) Stratified Adjudication Framework for assessing security awareness, execution behavior, and risk consequences. Extensive experiments show that current mobile agents are particularly vulnerable to implicit jailbreaks, as their safety alignment remains largely superficial and fails to generalize from conversational refusal to dynamic mobile interactions. Moreover, our dynamic evolution mechanism sharpens practical threat boundaries by continually uncovering new vulnerabilities, reaching a 100% ASR on Qwen3.5-122B. MobileJailBench provides a reproducible, continuously evolving framework for tracking the true security boundaries of mobile agents.