ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers
ShadowFPT shows how a malicious client in federated prompt tuning for frozen CLIP-style vision-language models can plant a targeted backdoor by learning a local “shadow trigger” that pushes triggered inputs toward a target class in the frozen representation space while uploading only normal-looking prompt updates. The neat bit is that it shifts the attack from suspicious prompt-parameter changes to input-side representation steering, making prompt-update anomaly defenses look inadequate and working across prompt types and federated settings, though the results seem to rely on the attacker having enough data/control to learn and adapt an effective trigger and are mainly demonstrated in the frozen-backbone FPT setup.
Abstract
Federated Prompt Tuning (FPT) adapts large vision--language models by freezing the pretrained backbone and optimizing only lightweight prompt parameters across clients. Although this design improves communication and parameter efficiency, it creates an overlooked security risk. Since the backbone is shared and frozen, a malicious client can induce backdoor behavior through the representation space while keeping its uploaded prompt updates close to benign ones. We propose \textbf{ShadowFPT}, a targeted backdoor attack that exploits this frozen-backbone attack surface. ShadowFPT first pretrains a learnable \emph{Shadow Trigger} against the frozen CLIP visual encoder, using either auxiliary public data or the malicious client's local data, so that triggered inputs are steered toward the target class in representation space. During federated prompt tuning, the malicious client adapts the trigger under the current global prompt and then optimizes its local prompt on both clean and triggered samples. Only prompt parameters are uploaded to the server, while the trigger and frozen encoders remain local. By shifting most of the attack burden from prompt updates to trigger-induced representation steering, ShadowFPT achieves targeted misclassification while preserving prompt-space stealthiness. Across multiple datasets, aggregation rules, and non-IID federated partitions, ShadowFPT increases the attack success rate from 19.81\% to 90.36\% in our main setting, while maintaining clean accuracy. It remains effective across textual, visual, and joint vision--language prompt tuning. These results identify frozen backbones as stealthy and underexplored backdoor surfaces in federated prompt tuning, suggesting that defenses based only on prompt-update anomaly detection are insufficient.