Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model
The paper audits a deployed developer-accessible on-device language model to see whether users or lightweight app developers can tell when it is wrong, finding that it often confidently confabulates on false-premise prompts, over-refuses benign ones, and reports confidence that is basically useless for discriminating correct from incorrect answers. What’s interesting is the “surface-indistinguishability” result showing confident-wrong and confident-correct outputs look nearly the same from user-visible signals, plus a practical black-box consistency wrapper that substantially improves reliability, though the study is still mostly a targeted audit of one deployable model/configuration and a bounded set of failure modes rather than a broad guarantee about on-device LMs in general.
Abstract
Aligning deployed language models requires knowing when their outputs can be trusted, yet on-device models now ship to hundreds of millions of devices with no server-side moderation, and the configuration developers can actually deploy is rarely audited independently. We present a reproducible reliability audit of the developer-accessible on-device foundation model, framed as an oversight question: can a user or a resource-constrained developer tell when the model is wrong? Red-teaming it on calibration, confident confabulation on false-premise questions, and over-refusal of benign prompts, we find a \emph{task-asymmetric miscalibration}: its guardrails fail in opposite directions across tasks (confabulating on 69\% of false premises while refusing 18\% of entirely benign inputs), atop a self-reported confidence that is saturated and non-discriminative (AUROC 0.47; ECE 70, worst among comparable small models). Crucially, confident-correct and confident-wrong outputs are \emph{surface-indistinguishable}: a classifier over 15 user-visible features separates them at AUROC only 0.55 (equivalence-confirmed), leaving no signal for oversight at inference time. No cheap single-generation signal flags these failures ($\le$0.68 AUROC), whereas a black-box consistency wrapper requiring no model access recovers reliability (confident confabulation 75\%$\to$3\%; selective accuracy 43\%$\to$83\%) at a tunable cost. We contribute a model-agnostic audit protocol, a surface-indistinguishability test, and released code and frozen evaluation items as reusable infrastructure for auditing deployed models.