Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time
The paper introduces an “influence score” for interpreting Transformer classifiers by measuring how individual attention heads affect prompt-injection detection decisions, combining their directional effect on output logits with their contribution to the residual stream across heads, layers, and the full network. Its nice angle is that it sits between detailed circuit tracing and coarse ablation/output analyses, giving a structured view of how correct and incorrect predictions differ, though the evidence sounds limited to one specialized DeBERTa prompt-injection detector and would need broader validation to show the score generalizes.
Abstract
We propose an influence score to quantify the contribution of attention heads to classification decisions in Transformer-based models designed for prompt injection detection. The score combines directional influence on the logits with structural contribution within the residual stream, enabling a multi-scale analysis at the head, layer, and network levels. Applied to a DeBERTa model specialized for prompt injection detection, our framework reveals distinct decision behaviours between correct and erroneous predictions. Our method provides an effective compromise between fine-grained circuit analysis and global output-based methods, and offers a systematic way to study decision mechanisms in Transformer classifiers.