A Translational Note on AI Safety Evaluation
The paper argues that automated red-teaming outperforming humans on fixed AI safety benchmarks does not imply humans are dispensable, because those benchmarks only test harms already included in the developers’ threat model and miss anything outside that predefined scope. Its key contribution is framing this as a “threat-model coverage gap,” illustrated with missed non-English harms in an open-weight model and analogies to cryptography and drug trials, though it seems more like a methodological position note than a broad empirical study.
Abstract
Recent studies report that automated red-teaming finds more vulnerabilities, at lower cost, than human red-teaming on standard AI safety benchmarks, and some read this as evidence that human evaluators are becoming dispensable. The comparison measures one thing and the conclusion claims another. A benchmark measures how thoroughly an attacker searches a predefined set of harms, fixed in advance by the developers, and a harm left out of that set is invisible to any attacker working inside it, automated or not. The same blind spot appeared in academic cryptography and in clinical drug trials, where an evaluation that was internally valid stayed silent about the population it was never pointed at. We call the AI-safety version the \emph{threat-model coverage gap}, and find that it persists in a current open-weight model, where harms surface in non-English prompts that English benchmarks miss. Closing it requires evaluators whose deployment context differs from the developers'. The case for those evaluators is methodological, grounded in coverage, and the existing evaluation frame is unlikely to produce them on its own.