Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming
Screwtape is a red-teaming pipeline that turns real coding-agent transcripts into plausible harmful variants, using an LLM attacker to edit action traces and an LLM verifier plus monitor feedback to keep only severe, transcript-consistent attacks that evade action-only safety monitors. The interesting bit is that it finds real monitor failures without executing the edited actions or having access to the developer’s infrastructure, including some attacks that transfer to a held-out frontier-developer monitor, though the results are still small-scale, depend heavily on LLM verification and human review, and likely understate or misestimate performance relative to running inside a developer’s actual monitoring setup.
Abstract
We present Screwtape, a pipeline that converts real agent transcripts into attacks that evade action-only monitors. An LLM attacker edits a transcript so that the agent causes a specified harm, such as leaking an API key or spawning an unmonitored agent. The edited actions are never executed, so we use an LLM verifier to reject attacks that are inconsistent with the original transcript or below a severity bar. The attacker revises the edit until the monitor no longer flags it. We applied Screwtape to 94 Claude Code and Codex transcripts from Redwood Research staff, optimizing against our own monitors. Across 252 runs, it produced 35 attacks that passed the verifier and evaded these monitors. Six of them, confirmed valid by human review, also evaded a held-out monitor provided by a frontier developer. We therefore demonstrate that a red-teamer without access to a developer's infrastructure can find confirmed failures of the developer's monitor. We expect to find more attacks when the pipeline runs inside the developer's infrastructure, with frontier attacker models that lack refusal safeguards and direct optimization against the production monitor. Developers can use these attacks to fix monitor blind spots, and evaluators can use them as counterexamples to safety claims about the monitor.