📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram
Originally published on software-engineer-blog.com.
You already know the trick: add "think step by step" to your prompt and the model's answer gets better. Almost nobody explains why — and the real reason has nothing to do with motivation or effort.
- Mental model: A transformer spends a fixed stack of layers per token, so adding reasoning tokens doesn't make the model smarter — it buys it more compute passes and an external scratchpad to read from.
The Problem: Fixed Compute per Token
Here's the floor. When a transformer generates a token, it runs through the same neural network layers every time. The stack depth is fixed at model-creation time. Whether you ask it "2+2" or "Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left?", the model gets the same amount of layered computation to produce each output token.
That compute budget never grows with problem difficulty.
Now imagine you ask for just the answer: "Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left? Answer only the number."
The model has to solve a three-step problem (multiply 3 × 8 = 24, multiply 4 × 5 = 20, subtract 24 − 20 = 4) in a single forward pass. It needs to hold "24" and "20" somewhere while computing the final step. But it's only got one forward pass, one set of layer outputs, and nowhere internal to stash intermediate values. So it guesses. It might say 19.
It didn't get the math wrong because it's bad at math. It got it wrong because you handed it the wrong compute budget for the job.
The Mechanism: Three Small Shifts
Now ask the same question and let it write the steps: "Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left? Think step by step."
Three mechanical things happen:
1. The model becomes a loop.
Every token the model emits is appended to the input context and fed back in on the next forward pass. So if it writes "First, 3 × 8 = 24", that token sequence gets read again when computing the next token. Write 40 reasoning tokens, run 40 forward passes instead of 1.
2. The written steps are external memory.
Instead of holding "24" in some internal embedding space and hoping it survives the next layer stack, the model writes "24" to the output — and then reads it back. The scratchpad is not hidden. It's right there in the text.
3. Each step conditions on prior work.
When the model generates "then 24 − 20", it's reading the output from step 1 ("3 × 8 = 24") and step 2 ("5 × 4 = 20") off the screen. No internal state decay. No lossy compression. Just tokens it can see.
Why It Helps Sometimes (and Not Others)
This prediction is tight: chain of thought helps when the problem has steps, and does nothing when it doesn't.
| Task Type | Why Steps Help | Why Steps Don't Help | |
|---|---|---|---|
| Multi-step math | Three multiplies and a subtract need three separate forward passes to avoid state collapse. Steps buy those passes. | — | |
| Logic / planning | Decomposing into sub-goals ("first find X, then use X to find Y") is steps. Writing them out reads them back. | — | |
| Single-fact recall | — | "What's the capital of France?" is one lookup, one forward pass. No steps exist. Writing fake steps adds latency and buys nothing. | |
| Categorization | — | If the model can classify an email as spam in one pass, additional "reasoning" tokens are just decoration. |
The pattern: if the ground truth solution has serial dependencies (step N depends on the output of step N−1), steps help. If it's a single lookup or one-pass classification, they don't.
The Honest Catches
The written reasoning is not an audit log.
The model can land on an answer and then construct a plausible chain of reasoning to justify it. The text reads like "here's how I thought," but it might be "here's a story that fits the answer I already generated." You're getting a rationalization, not a replay of the computation. This matters when you're trying to trust the intermediate steps or debug where a wrong answer came from.
It costs latency and tokens.
Each reasoning token is a forward pass. Forty tokens = 40× the inference latency compared to asking for just the answer. For simple tasks, you've made the model slower to get the right answer. For hard tasks, you've made it possible to get the right answer, and latency is the price.
Modern Context: Reasoning Models Changed the Game
This entire story was the standard explanation during the prompt-engineering era (2022–2024). But modern reasoning models (OpenAI o-series, Anthropic Claude extended thinking, Google Gemini) do chain of thought internally, in a hidden token stream that doesn't appear in your output. The model spends thinking tokens you don't see, then emits the final answer.
From the user's perspective: same model, same weights, same job. But now the reasoning is hidden and you pay for it upfront, not in your output tokens.
"Think step by step" is still mechanically sound — it still works — but it's largely a prompt-era artifact. Modern models abstract away the need to write it explicitly. If you're using one of those models, you're already paying for internal reasoning; adding "think step by step" to the prompt often does nothing.
For LLM Serving: Time to First Token vs. Time Per Output Token
If you operate an LLM serving layer, chain of thought reframes the latency budget:
- Direct answer (one token): TTFT (time to first token) + one TPOT (time per output token).
- Reasoning tokens (40 tokens): TTFT + 40 × TPOT.
The first pass is the same. But you're now serializing 40 forward passes instead of 1. If you're batching requests, the model could process other users' prompts during those intermediate passes — but not for this user. You've increased the time-in-flight for that request.
Reasoning models hide this cost by processing the hidden thinking tokens within the system's "thinking budget" before returning the final answer, but the wall-clock time is still spent. The difference: the user doesn't see the intermediate work, and you can potentially amortize batches better because the output is shorter.
Verdict
Reach for explicit chain of thought when you're solving multi-step math, logic, or planning problems on a base model and you can tolerate the latency cost. Reach for it not when you're doing single-fact recall or using a modern reasoning model — either the steps don't exist or the model already handles it internally.
Watch the 90-second reel for the same idea, compressed.
Top comments (0)