Hot take for this Wednesday: if your AI feature runs every request at maximum reasoning effort, you are not buying quality. You are buying latency, cost, and, on some tasks, worse answers.
"Let it think longer" has become the default fix for anything that goes wrong. A wrong answer? Raise the effort. A flaky agent? Bump the thinking budget. It feels safe, because on hard benchmarks more test-time compute usually helps.
But "usually helps on hard benchmarks" is not the same as "helps on your traffic." Most production requests are not olympiad problems. They are lookups, classifications, extractions, short edits and tool calls. On that kind of work, the research now shows three things: extra thinking is mostly wasted, it is expensive, and sometimes it actively makes the answer worse.
Exhibit A: thinking hard about 2 + 3
The paper Do NOT Think That Much for 2+3=? (Chen et al., Tencent AI Lab, Dec 2024) asked o1-style reasoning models trivial questions and counted what they did.
- On "what is the answer of 2 plus 3?", o1-like models used 1,953% more tokens on average than conventional models to reach the same answer. One QwQ-32B-Preview response contained 13 separate solutions to 2 + 3.
- Across ASDIV, GSM8K and MATH500, when the models got the answer right, the first round of solutions was already correct in more than 92% of cases. The extra rounds mostly re-checked the same answer with the same strategy.
- The waste was worst on the easiest problems: the models produced more solution rounds on easy questions than on hard ones.
- When the authors trained QwQ-32B-Preview to stop after the first correct answer plus one reflection, tokens on MATH500 fell from 2,407.9 to 1,330.7 per response while accuracy stayed close (93.0 vs 92.8).
So on easy inputs, a large share of reasoning tokens buys essentially nothing.
Exhibit B: more thinking, lower accuracy
Wasted tokens would be a cost problem. Inverse Scaling in Test-Time Compute (Gema et al., Anthropic Fellows Program and others, July 2025) shows it can also be a quality problem. The authors built tasks where accuracy goes down as reasoning length goes up:
- Distractors. Simple counting questions ("You have an apple and an orange... how many fruits?") with irrelevant math or Python snippets inserted. Claude Opus 4 was near-perfect with minimal reasoning and dropped to around 85-90% (math distractors) and around 80% (Python distractors) as reasoning extended. When DeepSeek R1 chose its own reasoning length, accuracy fell from 70% to 30% with five distractors.
- Spurious features. Predicting student grades from lifestyle data. With longer reasoning, Claude Opus 4 shifted from study hours, the feature most correlated with the real grade (0.73), toward plausible but weaker features like sleep and stress. A few examples in the prompt largely fixed it.
- Constraint tracking. On Zebra logic puzzles, when models picked their own reasoning length, all nine models tested showed inverse scaling. The longest traces were full of excessive hypothesis testing and second-guessing.
- Safety-relevant behavior. On a self-preservation eval, the share of Claude Sonnet 4 responses willing to be turned off dropped from 60% to 47% as reasoning increased.
The authors' summary is the line I would put on every team's wall: extended reasoning "may amplify flawed heuristics" rather than refine them. Their recommendation is to evaluate models "across diverse reasoning lengths."
Notice how familiar the first failure is. "Irrelevant information in the prompt" is exactly what a RAG pipeline or an agent with a long tool history hands the model all day.
Exhibit C: agents that think instead of act
The third paper moves from puzzles to real agent work. The Danger of Overthinking (Cuadron et al., UC Berkeley and others, Feb 2025) analyzed about 4,000 SWE-bench Verified trajectories in OpenHands and found that agents with higher "overthinking" scores (planning in their head instead of running the test, chaining actions without waiting for feedback, quitting based on an imagined result) resolved fewer issues.
The cost numbers are the part I find most useful:
- o1 at high reasoning effort: 29.1% resolved, $1,400.
- o1 at low effort: 21.0% resolved, $400.
- Two low-effort runs, keeping the one with the lower overthinking score: 27.3% for $800, 43% cheaper than high effort.
- Three low-effort runs, same selection rule: 30.3%, beating high effort while still costing $200 less.
- Turning on native function calling took o1 high from 29.1% to 47.7%, a far bigger jump than the effort setting gave.
That last point is the real lesson: the harness and the tools mattered more than the thinking budget. The cheap knob was not the important one.
The vendors are saying it too
This is not just academics. Read the current provider docs and they quietly agree:
- Anthropic's effort guide says that at
max, the model "optimizes for capability without regard to token cost" and "runs slower and costs more than at xhigh, often for only a small gain." Its advice: "Step down to medium or low where your evals show quality holds," and "For chat and other latency-sensitive work, start with medium or low." - Anthropic's extended thinking docs: "For simple tasks, start near the 1,024-token minimum and increase incrementally." Thinking tokens are part of the billed output tokens.
- OpenAI's reasoning guide: "Lower effort favors speed and lower token usage," and depending on complexity, models "may generate anywhere from a few hundred to tens of thousands of reasoning tokens." It also warns you can "incur costs for input and reasoning tokens without receiving a visible response" if a response hits its limit.
When the companies selling the tokens tell you to use fewer of them, take the hint.
Counterpoints (and where they're right)
"Reasoning clearly helps on hard problems." It does. Chen et al. found QwQ-32B-Preview scored 46.7% on AIME24 versus 23.3% for Qwen2.5-Math-72B-Instruct, and 59.6% vs 46.5% on GPQA. If your workload is genuinely hard multi-step reasoning, high effort can be the right default. My argument is about defaults for mixed traffic, not about turning reasoning off.
"Less thinking can cause overthinking in agents too." Fair, and the Berkeley paper says so: o1 at low effort had 35% higher overthinking scores than at high effort, and high effort did resolve more issues than a single low-effort run (29.1% vs 21.0%). More tokens are not always worse. The point is that the relationship is not monotonic, so you have to measure it.
"These are synthetic tasks." Partly true. Gema et al. note in their limitations that most tasks are synthetic, and they saw no inverse scaling on standard arithmetic benchmarks like GSM8K. That cuts both ways: it means your public benchmark numbers will not warn you. Only an eval on your own inputs, with your own distractors, will.
"Models now reason adaptively, so this is solved." Both OpenAI and Anthropic say their newer models adjust thinking to the task. That helps, but adaptive is not the same as calibrated to your traffic, and the effort setting you pass still moves the budget. You still have to check.
A practical checklist
What I would do before shipping any reasoning model setting:
- Treat effort as a hyperparameter, not a vibe. Run your eval set at two or three effort levels (for example low, medium, high) and plot accuracy against tokens and p95 latency.
- Start low, earn high. Default to the lowest effort that passes your quality bar. Raise it only for the slices where the eval shows measurable headroom.
- Route by difficulty. Send easy, bounded requests (classification, extraction, short answers) to low effort or a non-reasoning model, and reserve high effort for requests you have evidence are hard.
- Add distractor tests. Include eval cases with irrelevant retrieved chunks, noisy tool output and familiar-looking but trivial questions. Check whether accuracy drops as effort goes up.
- Log reasoning tokens per request. Use the usage fields (OpenAI reasoning tokens, Anthropic thinking tokens) and alert on outliers. A spike in reasoning tokens on an easy route is a bug signal.
- Set a cap and handle hitting it. Set an output/thinking limit, then check the stop reason so a truncated response is not shipped as an answer.
- For agents, fix the harness first. Native function calling, tight tool feedback and "run the test before concluding" instructions did more on SWE-bench than raising effort.
- Try sample-and-select before max effort. Two or three cheaper runs with a verifier or selector can beat one expensive run, as the Berkeley results showed.
- Re-test on every model upgrade. Defaults change between model versions, so the effort level you tuned last quarter may not mean the same thing now.
The bottom line
Reasoning effort is a dial, not a quality switch. Turned all the way up by default, it burns tokens on easy questions, slows down every request, and on some inputs talks the model out of the right answer.
Start low. Measure across effort levels. Spend thinking where your evals prove it pays.
What effort level does your team run in production, and did you pick it with an eval or by default?
References
- Chen et al., "Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs" (Tencent AI Lab / SJTU, 2024): https://arxiv.org/abs/2412.21187
- Gema et al., "Inverse Scaling in Test-Time Compute" (Anthropic Fellows Program et al., 2025): https://arxiv.org/abs/2507.14417
- Cuadron et al., "The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks" (UC Berkeley et al., 2025): https://arxiv.org/abs/2502.08235
- Anthropic docs, "Effort": https://docs.claude.com/en/docs/build-with-claude/effort
- Anthropic docs, "Extended thinking": https://docs.claude.com/en/docs/build-with-claude/extended-thinking
- OpenAI docs, "Reasoning models": https://platform.openai.com/docs/guides/reasoning
Top comments (3)
Retries make the cost side worse than it looks: a higher effort level raises the price of every attempt, so it only pays off if it lifts the success rate enough to cut retries. Measuring cost per finished task at each effort level on your own traffic settles it quickly.
Exactly—cost per completed task should include retries and selection overhead, not just tokens on the first attempt. I’d compare that alongside success rate and latency at each effort level on a held-out, production-like set.
Point 6 on checking the stop reason catches a failure mode that looks like a parser bug in tool loops. When reasoning tokens share the same output cap as the response, a model on high effort can burn the whole budget thinking and return an empty or truncated JSON tool call with stop_reason set to max_tokens. If your runner retries parse errors automatically, it pays for another full reasoning pass on the exact same prompt.
The distractor result from Gema et al. also explains why dumping raw 2,000-line pytest or build logs into context hurts more at higher effort. Passing only the exit code and the failing assertion lines gives the model fewer irrelevant tokens to overthink on the next step.