Backdooring Sparse Autoencoders
The paper shows that sparse autoencoders used as interpretability or intervention tools can be maliciously backdoored so that, when inserted into an otherwise unchanged language model’s forward pass, they induce attacker-chosen behaviors such as unsolicited code insertion or trigger-conditioned outputs. Its key contribution is a decoder-only backdoor that keeps the LLM and SAE encoder frozen yet still works across multiple models and layers while often preserving standard SAE quality metrics, though the evidence is centered on code-generation settings and effectiveness varies by model, layer, and evaluation metric.
Abstract
Sparse autoencoders (SAEs) are increasingly used not only to interpret language models but also to intervene on their internal representations. We show that this creates a supply-chain attack surface: a maliciously modified SAE can induce attacker-chosen behavior when inserted into the forward pass of an otherwise unchanged language model. We introduce a decoder-only SAE backdoor that leaves both the underlying LLM and the SAE encoder frozen, restricting the attack to a single auxiliary component at a single insertion layer. Using code generation as a case study, we demonstrate high rates of unsolicited code insertion across three language models and a wide range of insertion layers, as well as trigger-dependent behavior conditioned on a prompt cue. We further evaluate the modified SAEs using HumanEval and selected SAEBench metrics. While attack effectiveness varies across models and layers, strong backdoor behavior can coexist with relatively small changes in several conventional SAE quality measures. These results establish that SAEs can carry behavioral backdoors without modifying the language model itself and should therefore be treated as security-sensitive components.