What Actually Happens When a Transformer Thinks Step by Step: The Mechanics of Chain-of-Thought

You've seen it work. Add "Let's think step by step" to a prompt and the model suddenly solves math problems it couldn't touch before. The output looks like reasoning — intermediate steps, logical progression, a final answer. But what's actually happening inside the transformer?

It's not magic. It's not the model "thinking" in any human sense. It's a concrete computational mechanism: written tokens buy serial depth.

Here is what actually happens.

The Core Constraint: Fixed Serial Depth

A transformer decoder has L layers. In one forward pass, information flows from the input embeddings up through layer 1, then layer 2, all the way to layer L. The longest dependency path has exactly L steps.

This means: within a single forward pass, the model can only perform L rounds of "compute something, then use the result." Width doesn't help — you can have billions of parallel multiplications per layer, but you still only get L sequential steps.

Some problems need more sequential steps than L. Multi-step arithmetic, long-division, composing twenty permutations, tracing a path through a tree — these have serial depth D > L. One forward pass literally cannot solve them, no matter how wide the model.

How CoT Breaks the Limit

Generation changes the computation graph. When the model outputs token t, that token gets fed back into layer 1 at position t+1. Now there's an edge going down from the top of the stack back to the bottom.

Pass 1: input → layer 1 → ... → layer L → token 1
Pass 2: token 1 → layer 1 → ... → layer L → token 2
Pass 3: token 2 → layer 1 → ... → layer L → token 3
...

After T generated tokens, the longest path has L × T edges. Every written token buys one more full pass through the network. The written text is the model's working memory — and it's the only channel (in standard decoders) for information to travel from the output of one pass back to the input of the next.

The KV cache doesn't change this. It saves recomputation of keys and values, but layer ℓ still only reads from layer ℓ-1. The cache saves time, not depth.

What the Model Is Actually Doing With Those Tokens

Three mechanistic insights from recent research explain what those intermediate tokens are doing:

1. Decoding Space Pruning (The Template Effect)

When a model generates a CoT, the probability distribution over next tokens becomes sharper and more concentrated at each step. The intermediate steps act as answer templates — they constrain what tokens are plausible next.

Research on LLaMA-3.1-70B shows CoT prompts consistently produce higher-probability, lower-entropy distributions than direct-answer prompts. The model isn't "reasoning" — it's following a structural groove that the prompt examples carved into its prediction space. Higher template adherence correlates strongly with better performance.

2. The Functional Rift: Two Halves, Two Jobs

Mechanistic analysis of Llama-2 7B on fictional reasoning tasks reveals a sharp phase shift at the middle layers (~layer 16 of 32):

First half (layers 1-16): Attention heads move information between ontologically related entities. They copy "numpus" → "rompus" so patterns transfer. Representations stay close to pretraining priors (bigram associations).

Second half (layers 17-32): Attention heads write answer tokens to the residual stream. Multiple heads do this in parallel, collecting answers from the question context, few-shot examples, and the generated CoT itself. The in-context prior takes over.

The model literally switches algorithms halfway through: first it aligns representations to the problem structure, then it harvests answers from the context it just built.

3. Iteration Heads: The Circuit for Serial Computation

In controlled experiments on iterative tasks (parity, copying, polynomial iteration), transformers develop a specialized two-layer attention circuit — iteration heads — that implements a state machine:

• Head 1 (layer 1): Finds the "end of input" token, extracts its positional encoding
• Head 2 (layer 2): Uses that position to retrieve the t-th input element and the previous state s_{t-1} from the residual stream
• MLP (layer 2): Computes the next state s_t = F(s_{t-1}, x_t)

This circuit is task-agnostic — the same attention pattern solves copying, parity, and polynomial iteration. Only the MLP changes. Fine-tuning just the second-layer MLP transfers the skill between tasks in <20 epochs.

Real LLMs likely develop analogous "inner circuits" for multi-step reasoning, trained on the vast amount of iterative reasoning in code and natural language.

Neuron Engagement: Task-Dependent Modulation

CoT doesn't just "use more neurons." It modulates neuron activation differently depending on the task type:

• Open-domain tasks (GSM8K, SVAMP): CoT reduces average neuron activation (~790K vs ~820K in LLaMA-3.1-70B). The structured steps let FFNs operate more selectively.
• Closed-domain tasks (AQuA multiple choice): CoT increases neuron activation. The constrained answer space requires more active feature matching.

The model isn't working harder — it's working differently, allocating compute where the task structure demands it.

The Reliability Trade-off

Every written token is a chance to slip. At 0.1% error per token, a 2,500-token chain survives only 8% of the time. This is why:

• Reasoning models train for self-repair (backtracking, checking)
• Terse states help (fewer tokens = fewer failure points)
• Sampling multiple chains and voting (majority vote) dramatically improves reliability

The chain isn't free compute — it's compute with compounding risk.

What This Means for You

Misconception	Reality
"CoT makes the model smarter"	CoT gives the model more serial steps — it spends existing capacity differently
"The written steps are the reasoning"	The steps are the record of reasoning; the actual computation happens in the forward passes between tokens
"Longer chains are always better"	Longer chains buy depth but multiply error probability. Optimal length balances depth against reliability
"CoT works for everything"	CoT helps on problems with high serial depth (math, logic, code). It doesn't help on fact lookup or pattern matching

The Mental Model

Think of a transformer as a fixed-depth circuit that can be recursively applied. CoT is the mechanism that enables recursion: each output token becomes the next input, letting the circuit run again on an expanded context.

The "reasoning" you see in the output? That's the trace of the circuit's intermediate states, written into the only memory the model has: its own context window.

Next time you add "Let's think step by step," you're not asking the model to think. You're giving it permission to run its circuit more times.