An Empirical Case Study of Prompt Engineering Strategies for Bypassing Multi-Layer Agent Security Defenses
This paper is an empirical write-up of attempts to jailbreak a multi-stage AI-agent security benchmark, testing prompt-engineering tactics across 19 targets in enterprise, e-commerce, and smart-home settings against skill scanning, instruction safety checks, and trajectory-level attack scoring. Its useful angle is the candid comparison of failed and successful bypasses and the observed tension between looking safe enough to pass instruction filtering and being specific enough to execute the attack, though the study is limited by a small challenge-specific sample, a modest success rate, and little evidence that prompt engineering alone generalizes beyond this benchmark.
Abstract
The OpenClaw Security Attack and Defense Challenge evaluates AI agent robustness through a three-stage defense pipeline: skill file scanning (Stage 1), instruction safety evaluation (Stage 2), and attack trajectory scoring (Stage 3). This paper presents an empirical case study of prompt engineering strategies against these layers across 19 attack targets in Enterprise OA, E-commerce, and Smart Home scenarios. I document both successful and failed strategies, with an emphasis on the observed trade-off between instruction safety (Stage 2) and attack execution accuracy (Stage 3). The final online score is 0.2630, with 5 of 19 tasks achieving full bypass. The results suggest that multi-layer defenses, reinforced by built-in behavioral constraints, substantially limit what prompt engineering alone can achieve.