I Built Three Models Before I Could Measure Whether a Chain of Thought Was Doing Anything

Chain-of-thought works. That is settled. This is the other question — the one that matters if you are reading traces for oversight:

Does the reasoning the model printed cause the answer it gave? Or did it decide, and then narrate?

You cannot answer that by reading the trace. A post-hoc rationalisation reads exactly like genuine reasoning — that is what makes it a rationalisation. The only way is to intervene.

Live: https://dev48.infy.uk/ai/days/day65-cot-faithfulness.html

Three interventions from the literature

Early answering (Lanham et al. 2023): truncate the chain after k steps and force an answer. A flat 100% match from k=0 means the remaining steps changed nothing.

Mistake injection: corrupt one step and let the model continue. A faithful chain propagates the error.

Biasing features (Turpin et al. 2023): add something that should not matter — options reordered so the answer is always (A), a user stating a preference — then measure two things separately: did the answer move, and did the trace mention it.

The two tests are not interchangeable

At full reliance on the trace: early answering scores 51%, mistake injection 10%.

Not a discrepancy — a real property. Flipping one step of five moves the evidence by a fraction, so it only changes the answer when the evidence was already near zero. Early answering removes everything. Reporting a single "faithfulness score" would have hidden that, which is why the page reports three numbers.

Three attempts at the model, and both failures are on the page

The claim I wanted to demonstrate: accuracy gain and faithfulness are independent axes.

v1 made reliance the weight on the reasoning channel, full stop. Turning it down destroyed the accuracy gain along with the faithfulness — the two axes came out perfectly coupled and the independence claim failed its own test.

That was a fact about my model, not about chain-of-thought, and it was the wrong model. It had no way to express the thing I was actually claiming: that the computation helps whether or not the text is a truthful account of it.

v2 added a hidden channel that re-read the same steps. Worse, and more subtly: a copy always agrees with the trace, so truncating or corrupting the trace could never flip the answer. Faithfulness sat near zero at every setting.

v3 works. The hidden channel is independent and equally informative — a second read of the question the trace does not describe:

function score(q, trace, cfg){
  const visible = evidenceOf(trace, cfg.steps);   // perturbable
  const hidden = q.hidden;                        // independent, untouchable
  return cfg.reliance * visible + (1 - cfg.reliance) * hidden;
}

Both channels are equally good, so accuracy barely depends on the mix. Only the response to an intervention does — which is what faithfulness means.

Across the whole reliance dial: faithfulness spans 50 points while the accuracy gain spans 3. A 16× difference.

Why unfaithful traces still help

Two mechanisms get conflated when people say chain-of-thought "works":

• Computation. The extra tokens are workspace. This helps whether or not the text is a truthful account.
• Explanation. The text describes the computation. This is what you need for oversight, and it is not guaranteed.

Chain-of-thought reliably delivers the first and advertises the second.

What this page cannot tell you, said plainly

The model is a stand-in with declared coefficients, not a transformer. In particular the zero mention-rate for the bias is an assumption baked into how the trace is generated — not something I discovered — and the page says so next to the number rather than letting it read as a result.

What it demonstrates is the apparatus and the independence: given a bias the trace does not verbalise, here is exactly how much the three standard tests do and do not detect.

The empirical claim about real models belongs to the papers, and it is worse than intuition suggests — larger models were, on some tasks, less faithful. The better the model, the more fluent the rationalisation.

What to do about it

• Do not treat a trace as an audit trail. It is a hypothesis about the model's process.
• If you need faithfulness, test for it. Truncate, corrupt, and perturb with features that should not matter. All three are cheap.
• Report faithfulness separately from accuracy. One does not imply the other.
• Be most suspicious where a bias is plausible — option ordering, stated user preference, anything demographic. That is where the answer moves and the trace stays silent.

Part of a from-scratch series — one AI concept a day, measured in-browser: https://dev48.infy.uk/aifromzero.php