Does On-Policy Distillation for Safety Pose Backdoor Risks?
The paper shows that using on-policy distillation to make a student LLM safer can also silently transfer a backdoor from a safety-aligned but malicious teacher, with surprisingly small poisoning rates producing high trigger-conditioned harmful behavior in the student. Its useful insight is that seemingly routine OPD choices like more epochs and top-k KL rewards can amplify backdoor transfer, while a simple KL-clipping “Lazy Defense” can slow it down, though the mitigation is limited and the work mainly demonstrates the risk rather than providing a robust fix.
Abstract
On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat model, a poisoning rate as low as 3% results in an attack success rate (ASR) of up to 70% on the distilled student. We further identify two training choices that can amplify this risk. First, increasing the number of training epochs can lead to high ASR even at low poisoning rates. With only 10 poisoned samples, ASR reaches 67% after 16 epochs. Second, the commonly used top-k KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings. Alongside these findings, we explore a simple mitigation, Lazy Defense, which clips KL rewards to make student updates less aggressive, limiting aggressive updates and slowing backdoor learning. Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings. Together, our findings reveal that OPD can propagate backdoors, highlighting the need to address the safety risks of OPD.