Defensive Sufficiency in a Stackelberg Model of AI Security
The paper analyzes when iterative AI security feedback loops—automated testing, red teaming, incident response, and repair—eventually cover an attack surface, then embeds that process in a defender-led Stackelberg game where the defender chooses discovery and repair investments anticipating attacker search effort. Its useful twist is separating eventual per-attack protection from being fully protected at any given time and tying that to economic deterrence regimes, though the model leans on strong assumptions like effective non-regressing repairs, persistent discovery probabilities, and fairly stylized attacker/defender behavior.
Abstract
Feedback from automated testing, human red teaming, and incident response can strengthen an AI system's defenses when discovered failures lead to effective repairs. We study when this feedback process provides sufficient protection and when investing in it is economically worthwhile. We begin by showing that an attack surface composed of finite number of inputs is defended with probability 1 if every unresolved attack has a persistent chance of discovery, repairs are effective, and subsequent updates preserve earlier protection. We derive completion-time bounds and extend the analysis to growing attack surfaces, repairs that generalize across related attacks, and multiple discovery mechanisms. These results distinguish eventual protection against each fixed attack from complete protection at a single time. We then formulate a defender-led Stackelberg game in which the defender invests in proactive discovery and reactive repair, anticipating the attacker's choice of search effort. We characterize the least-cost allocation that deters attack and the equilibrium regimes in which the defender funds neither capability, one capability, or both. Numerical experiments illustrate these regimes and show how faster repair can reduce compromise duration without reducing compromise probability.unified theory of performance limits in generative language models.