Alignment via Training Against Probes Without Losing Monitorability
The paper studies aligning language models by fine-tuning directly against probes trained to detect undesirable internal activation patterns, evaluating linear/nonlinear probes across harmlessness and honesty instead of relying only on output-based signals like preferences or rewards. The key insight is that fixed probes are easy for models to game, but continuously updated probes can improve safety and honesty while preserving utility and leaving the relevant concepts linearly monitorable, though the approach still depends on having probes that actually capture the behaviors of interest and is only demonstrated on a limited set of alignment targets.
Abstract
Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety-utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.