The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls
The paper tests multilingual chain-of-thought faithfulness on parallel MMMLU multiple-choice questions and finds that, for knowledge-heavy tasks, models’ answers depend more on their CoT in lower-resource languages, while reasoning-heavy tasks do not show the same resource-level trend. The interesting bit is the “Reliance Hypothesis”: when the model can’t just recall an answer, it leans more on generated reasoning, with confidence producing a Simpson’s-paradox-like pattern; the main caveat is that the evidence is still correlational and limited to eight languages, four models, and MMMLU-style multiple-choice settings.
Abstract
Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English. Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on their reasoning. To our knowledge, we are the first to relate CoT faithfulness to both resource level and task type (knowledge- or reasoning-intensive). On parallel multiple-choice questions from Multilingual Massive Multitask Language Understanding (MMMLU) in eight languages, faithfulness rises as resource level falls on knowledge-intensive tasks but not on reasoning-intensive ones, a gap that points the same way across four models and several faithfulness measures. We propose, and find support for, the Reliance Hypothesis: models rely more on their reasoning when they cannot recall the answer, as in lower-resource languages on knowledge-intensive tasks. Model confidence partly explains the effect of resource level, but not the knowledge-reasoning gap. Languages in which models are less confident rely more on reasoning, yet less confident answers within a language do not, a Simpson's paradox. In lower-resource languages, models rely on their reasoning for more answers, not more strongly for each. CoT monitoring may therefore be most informative in lower-resource languages, where it is least often validated.