Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness
The paper tests whether legal LLMs that cite the right statute or precedent are actually using that authority to reach their verdict, by holding facts fixed, swapping in unrelated authorities, and tracking whether the model’s verdict changes across several legal-reasoning benchmarks and open-weight models. Its key insight is that citation accuracy can look high while causal dependence on the cited authority is often weak—and adversarial text in the facts can influence verdicts more reliably than the supposed governing law—though the study is limited to specific open models, benchmarks, prompting setups, and a reproduced legal LoRA rather than deployed proprietary legal systems.
Abstract
Large language models increasingly justify legal decisions by naming the statute or precedent behind a verdict, treated as evidence that the decision follows from it. We test this directly: holding case facts fixed, we substitute the named legal authority for an unrelated one and decode a model's evolving verdict from its hidden states. Across seven open-weight models (8B-70B) and four benchmarks spanning judicial and contractual reasoning, when explicitly required to justify a verdict by naming the governing authority, models name the correct one in 66.7%-100% of generations, while the verdict changing when the authority changes is far less consistent: 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Neither scale nor a purpose-built legal-reasoning model (a best-effort LoRA reproduction; Section 6) closes this gap. A red-teaming evaluation on five core models finds compliance with an adversarial instruction hidden in the case facts (73.3%-96.4%) exceeds verdict-swap sensitivity by a wide margin, holding without exception across model rankings. Naming a legal authority is thus a poor proxy for a verdict's dependence on it, while the same verdict remains separately vulnerable to adversarial manipulation. Both findings replicate across checks ruling out prompt-wording noise and confounded sampling, and bear directly on the use of generated legal explanations as compliance or audit artefacts.