Monitor-Validity Failures: LLM Safety Evidence Paths at Evolving Framework Boundaries
The paper argues that LLM safety failures can occur not because a detector misses bad behavior or a jailbreak succeeds, but because already-detected safety evidence gets lost or disconnected as it crosses framework boundaries like tool-call adapters, streams, verifier inputs, or policy-enforcement paths. Its useful contribution is a concrete “canary contract” method for regression-testing these evidence paths across real framework repairs, surfacing 18 failures in several ecosystems; the main caveat is that it’s repair/audit driven rather than a broad prevalence study, so it shows this failure mode is real and under-tested more than it quantifies how often it happens in deployed systems.
Abstract
Modern LLM applications rely on runtime monitors that inspect tool calls, traces, streams, detector outputs, and verifier inputs. This creates an interface-reliability problem: a framework path may remain operational while policy-relevant evidence never reaches its decision point. In one representative case, a regex detector returned a positive match, but the framework stranded the result before the block decision. This differs from jailbreaks and detector errors because the evidence is already recognized yet disconnected from enforcement. We study this repair-boundary signature with canary contracts: parent/fixed confirmation units that place a known payload only on the changed evidence surface and record monitor visibility together with the downstream decision for action-boundary cases. Applying the protocol to public framework repairs, we confirm 18 evidence-path failures across 11 LLM or agent safety project groups and four implementation ecosystems. Nine are established-boundary repairs, eight are new-surface policy-coverage gaps, and one is a boundary-only evidence case. Fourteen reach E1 enforcement boundaries: seven native action outcomes, six configured enforcement boundaries, and one synthetic-consumer consequence. The remaining cases affect verifier, safety-prompt, or observability channels. The cases form carrier, adapter, authority, and surface families and show how ordinary application behavior can remain intact while a safety-specific evidence path misses its decision. Exact revisions, deterministic outcomes, negative controls, and a frozen prospective audit document the confirmation rule. Monitor assumptions therefore need regression tests at the schemas, streams, tools, and adapters that frameworks evolve.