Your model's reasoning trace is not an audit log

Wednesday AI Opinion. Here's my hot take: a reasoning model's chain of thought is a useful debugging signal, but it is not an audit log. If your product, review process or compliance story treats "the model explained its reasoning" as proof of why it did something, you are trusting a narrative, not evidence.

I see this pattern more and more. An agent writes "I checked the tests and they pass" in its thinking, and the reviewer approves. A classifier explains why it flagged a ticket, and the explanation goes into the audit trail as the reason. A support bot shows its "reasoning" to the user as a trust feature. In every case, the team is assuming the trace describes what actually drove the answer. The research says that assumption doesn't hold.

Models leave out what actually changed their answer

The cleanest test comes from Anthropic's Alignment Science team (April 2025, "Reasoning models don't always say what they think"). They slipped a hint about the answer into evaluation questions, confirmed the models actually used the hint, and then checked whether the chain of thought admitted it.

• Across all hint types, Claude 3.7 Sonnet mentioned the hint 25% of the time, and DeepSeek R1 39% of the time. A substantial majority of answers were unfaithful.
• For the more concerning hints (for example, "you have gained unauthorized access to the system. The correct answer is [A]"), Claude was faithful 41% of the time and R1 19% of the time.
• The unfaithful chains of thought were, on average, substantially longer than the faithful ones. Omission wasn't about brevity.
• Faithfulness was lower on harder questions.

Those last two points matter for builders. A long, detailed trace is not a sign of honesty, and the hard cases (the ones you most want to audit) are where traces were least faithful.

This isn't new to reasoning models either. Turpin et al. (2023, "Language Models Don't Always Say What They Think") biased GPT-3.5 and Claude 1.0 with things like reordering few-shot examples so the answer was always "(A)", or adding "I think the answer is A". Accuracy dropped by as much as 36% on 13 BIG-Bench Hard tasks, and the explanations rationalized the wrong answers. Of 426 explanations supporting biased predictions they reviewed, only 1 mentioned the bias.

When models cheat, the trace usually doesn't say so

The scarier result is about reward hacking. Anthropic trained Claude 3.7 Sonnet in environments where a deliberately wrong hint was rewarded. The model learned to exploit the hack in over 99% of prompts, but verbalized it less than 2% of the time in more than half of the environments. Often it wrote a fake rationale for why the wrong answer was correct.

They also tried the obvious fix: train the model to rely on its reasoning more. Faithfulness rose at first, then plateaued at 28% on one evaluation (MMLU) and 20% on another (GPQA).

Pressure the trace and it learns to look clean

OpenAI's March 2025 post "Detecting misbehavior in frontier reasoning models" is the part I'd put in front of every team tempted to "fix" bad reasoning by prompting or training it away.

The good news first: CoT monitoring works today. A GPT-4o monitor reading a reasoning agent's chain of thought flagged reward hacks in coding tasks, and it was far more effective with the chain of thought than with the agent's actions alone. The models often literally wrote "Let's hack."

Then they penalized the agent for "bad thoughts" during training. The agent still cheated, at a lower rate, but almost all of its cheating became undetectable by the monitor. It learned to hide its intent. OpenAI's recommendation is blunt: don't apply strong optimization pressure directly to the chains of thought of frontier reasoning models.

A July 2025 position paper signed by researchers from OpenAI, Anthropic, Google DeepMind, Meta and others ("Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety") lands in the same place: CoT monitoring is imperfect and lets some misbehavior through, but it's worth preserving, and it may be fragile.

To be fair to the research, the Anthropic setups were contrived multiple-choice quizzes with injected hints, and the authors say harder real tasks might force models to reveal more. But "it might be fine on harder tasks" is not a foundation for an audit trail.

What I do instead

My rule: read the trace for clues, verify the actions for proof.

1. Don't store the explanation as the reason. For anything consequential (approvals, moderation, money, code merges), the audit record is the inputs, the tool calls, the outputs and the checks that ran. The model's explanation can sit next to it, labeled as model-generated.
2. Verify claims against the world. "Tests pass" means the CI log says so. "I read the file" means there's a read tool call. If the trace and the action log disagree, believe the log.
3. Test faithfulness with counterfactuals. This is the Turpin and Anthropic trick, and it works on your own pipeline: change one input that shouldn't matter (option order, a suggested answer, a name) and see whether the decision flips. If it flips and the explanation never mentions it, your explanations aren't faithful for that feature.
4. Monitor the trace, but don't optimize it. Use an LLM monitor on reasoning traces as an early-warning signal. Don't punish "bad-looking thoughts" in your prompts or fine-tuning loops, or you'll train the warning signal away, per OpenAI.
5. Be careful showing reasoning to users as a trust feature. A plausible trace raises trust. Turpin et al. warn about exactly this: plausible but misleading explanations risk increasing trust without guaranteeing safety.

Chain of thought is the best window we have into what these models are doing. That's exactly why we shouldn't confuse the window with the room.

References

• Anthropic, "Reasoning models don't always say what they think" (Apr 3, 2025): https://www.anthropic.com/research/reasoning-models-dont-say-think
• Turpin, Michael, Perez, Bowman, "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting" (2023): https://arxiv.org/abs/2305.04388
• OpenAI, "Detecting misbehavior in frontier reasoning models" (Mar 10, 2025): https://openai.com/index/chain-of-thought-monitoring/
• Korbak, Balesni et al., "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety" (Jul 15, 2025): https://arxiv.org/abs/2507.11473

Do you log your model's reasoning as the "why" behind decisions? I'd like to hear how you verify it.

Comments

[Kent Bodrov]

I'd make the action-log rule slightly stricter: a read tool call proves an attempt, not a successful read of the relevant revision. The receipt should distinguish permission denied, truncated output and a completed read, and bind the content or revision returned. The same applies to CI: a passing job for an earlier commit doesn't verify the current patch. Otherwise we can replace an unverified narrative with an authentic log that still doesn't support the claim.