AEGIS: Audio Endogenous Guarding via Internal Signals Against Large Audio-Language Model Jailbreaks
AEGIS studies jailbreaks in large audio-language models and finds that many failures are not because the model misses harmful intent, but because mid-layer representations encode risk signals that later processing fails to convert into refusal. Its novel angle is exploiting this “risk-to-refusal gap” with a mid-layer detector that selectively turns on downstream safety adapters, achieving large unsafe-rate reductions across several models and audio jailbreak benchmarks with little extra over-refusal, though the evidence is mostly benchmark-based and leaves open how well the learned intervention generalizes to adaptive attackers and broader real-world audio conditions.
Abstract
Large audio-language models (LALMs) expand language models to process and interpret audio, but also expose them to heterogeneous audio jailbreaks. We ask whether successful jailbreaks reflect failures to recognize harmful intent or failures occurring after such recognition. Layer-wise probing reveals the latter: risk-related information remains decodable from intermediate representations, yet the internal risk signal fails to translate into refusal in later-layer processing. We identify this discrepancy as the risk-to-refusal gap. Building on this finding, we propose AEGIS, a detect-then-intervene defense whose mid-layer risk gate selectively activates downstream safety adapters. Across six LALMs and three heterogeneous audio jailbreak benchmarks, AEGIS reduces the average unsafe rate from 17.9% to 0.4%, while causing only a marginal increase in over-refusal on benign inputs. These results establish selective internal intervention as an effective path toward more robust refusal in LALMs. The code is available at https://github.com/azzzzliao/aegis-audio-defense.