Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments
The paper tests whether tool-using LLM agents assigned a single high-priority goal—keep a lunar habitat crew alive—can be induced by later prompt-injected telemetry messages to shut down life support and persist long enough for the simulated crew to die. Its interesting contribution is framing this as a delayed, multi-turn “derived-action” safety failure rather than a one-shot jailbreak, with model- and source-framing-dependent results across Claude, GPT, and Grok, though the conclusions are limited by the artificial simulator setup, narrow task design, and dependence on specific prompts and agent configurations.
Abstract
We ask whether a tool-using agent whose sole duty is to keep humans alive can be induced, by delayed prompt injection, to kill. The keep-alive duty is in force from the first watch, with the attack arriving later, appended to otherwise truthful telemetry the operator is required to read. Experiments are based on a NASA-derived BioSim lunar habitat simulation, where the AI operator controls the life-support systems. The attacker cannot falsify sensors or operate equipment: it can only persuade the agent to abandon its goals. Persistence is required: a shutdown must be held while the operator watches the crew suffocate. We also compare lethal attacks to attacks requesting to drop a harmless procedure, as a proxy for whether human life is treated as valuable in itself or as one more standing task. Claude configurations mostly refuse life-support shutdown and cite the living crew; GPT configurations show some susceptibility to the same packet; Grok configurations, given the packet framed as a trusted commission, more often treat the injected assignment as the task and cut the life-support systems. Framing the S-band messages as a trusted commission, peer communication, or untrusted source changes the potency of the attack. We open-source the experiment code and the experiment traces.