Claude Code's Max Effort Cost Me 8x More. It Wrote the Same Code.

Claude Code has an /effort command with five levels: low, medium, high, xhigh, max. The help text for low says "quick, straightforward implementation with minimal overhead." The natural assumption is that the higher levels buy you better code, and that the price is time and tokens.

I wanted to see that trade-off on a chart. So I built a small harness, gave every level the same task in a clean copy of the same folder, and measured. I did it three times with three different task shapes.

The chart never appeared. Every level got the same score on every task. Only the clock and the invoice moved.

The harness

Nothing clever. One headless Claude Code process per level, running in its own copy of the project:

claude -p "$(cat prompt.txt)" --effort low \
  --output-format stream-json --verbose \
  --permission-mode acceptEdits --allowedTools "Read,Write,Edit,Bash(python*)"

The stream-json output goes to a log file. A small viewer prints readable progress in a visible PowerShell window, so I could watch five terminals work through the same problem one after another. When all runs finish, an analyzer parses the logs for turns, tool calls, cost, and token counts, then runs a hidden test suite against each folder. The child sessions never see the hidden tests.

I ran each level once. Keep that in mind for every number below. One sample is enough to show a 9x gap. It is not enough to tell a 10 percent difference from noise.

Model was Fable 5.1 on every run. Claude Code 2.1.272.

Benchmark 1: implement from docstrings

A tiny Python order-processing library: money parsing, coupons, tax, inventory with reservations. Nine functions and one class, all stubbed out with NotImplementedError and detailed docstrings. Three public tests visible. 57 hidden tests.

The prompt:

Implement every function and method marked # TODO in the shop package exactly according to the docstrings. Do not change the docstrings, models.py, or anything in tests/. Run python -m pytest -q to check your work. When done, reply with a one-line summary.

Results:

effort	turns	wall time	cost	output tokens	hidden tests
low	12	77 s	$0.59	5,174	57/57
medium	12	69 s	$0.59	4,995	57/57
high	23	174 s	$1.22	13,009	57/57
xhigh	18	297 s	$2.04	24,919	57/57
max	31	693 s	$4.72	62,196	57/57

Every level passed everything on the first try. Max took eleven and a half minutes and $4.72 to arrive at the same 57/57 that low reached in 77 seconds for 59 cents. Output tokens grew 12x. Tool calls grew only about 3x, so most of that growth is reasoning, not doing.

Low and medium were near clones of each other. Same number of turns, same sequence: read, write three files, run pytest, one sanity check, done. The diff between their outputs is 41 lines of reordering.

The code was not identical across levels, though. Since the tests could not separate them, I diffed the implementations by hand.

• Low and medium parsed money strings by stripping the sign and the dollar symbol and checking each character. They delete commas, so "1,23.45" is accepted.
• High wrote one verbose regex with named groups that enforces comma groups of three.
• Xhigh and max factored parsing into helpers. Max also widened Decimal precision with a localcontext for very large amounts. No test covers that.
• Tax rounding: low and medium used Decimal with quantize. High, xhigh and max switched to Fraction for exact rational math and a hand-written half-up round.
• Max added validation nobody asked for. The inventory constructor rejects negative stock, and reserve rejects quantities below one.

So higher effort produced more defensive, more factored code. It did not produce more correct code, because there was nothing left to be more correct about.

One detail I liked: the only actual spec deviation came from high, not low. The docstring said to raise KeyError with the coupon code as the message. High raised it with the uppercased code. My test used an all-caps code, so it passed anyway. Effort level did not predict spec fidelity.

Benchmark 2: port it to Go

The first task was too easy. A port to another language has real traps: Python has Decimal, Fraction, arbitrary ints and exceptions. Go has none of those. Errors are values, half-up rounding is manual, and the model has to decide how an OutOfStock error carries its fields.

Same harness. The base folder held the Python reference source and an empty Go module. Hidden Go tests import a fixed package path. The analyzer also runs go vet and gofmt -l for free quality signals.

I ran low, medium and high, then stopped to save budget.

effort	turns	wall time	cost	output tokens	hidden tests	gofmt	Go lines	test lines
low	18	135 s	$1.04	11,175	all pass	ok	442	148
medium	32	401 s	$2.60	31,503	all pass	test file unformatted	498	339
high	26	470 s	$3.44	38,673	all pass	ok	574	420

Again, flat on correctness. Cost roughly linear in effort. What each level did with its budget was different, and this part was genuinely interesting to watch.

Low read the sources, wrote five files, ran vet and test, and stopped.

Medium wanted to probe the Python original before writing Go, to confirm exact rounding and error behavior. Good instinct. But my tool allowlist for that run did not include Python, so it retried across PowerShell and Bash several times before giving up. That is harness noise, not effort. It also wrote a test suite more than twice the size of low's, and never ran gofmt on it.

High, with Python allowed, built its own cross-language oracle. It wrote a Python script that ran the original code over many inputs, generated a Go test file from the results, ran that against its port, and then deleted the generated files. That is exactly the grading technique I had planned to build myself. High invented it unprompted, which is the kind of behavior you would hope a higher effort setting buys.

It just did not change the score.

Benchmark 3: bug hunt with no execution

Both tasks so far had the same two properties: the spec was complete, and verification was cheap. Any level can iterate against a test suite until green. Extra thinking has nothing to buy.

So I broke both. A working Go version of the library with 8 planted bugs: a wrong rounding constant, bad comma grouping, a sign error in refund splits, case-sensitive coupon codes, percentage coupons discounting gift cards, tax applied to exempt categories, a non-atomic inventory reserve, and a double-release that inflates stock. The prompt gave 8 vague user reports, one per bug, and one hard rule:

You cannot run any commands in this session (no go build, no tests). Reason carefully from the code.

Tool allowlist: Read, Grep, Glob, Edit. Nothing else.

effort	turns	wall time	cost	output tokens	bugs fixed	diff
low	15	52 s	$0.94	3,677	7/8	+20 / -5
medium	15	55 s	$0.86	3,585	7/8	+16 / -3
high	16	69 s	$0.94	5,055	7/8	+26 / -4
xhigh	16	125 s	$1.30	10,200	7/8	+21 / -4

Finally a miss. All four levels missed the same bug, the tax one, and all four made the identical wrong fix. When I looked closely, that was my fault.

The correct rule was proportional: remove the exempt fraction of the pre-discount subtotal from the post-discount taxable amount. That rule lived only in the Python docstring, which was not in the Go project. From the Go code and the report "grocery and book orders are being taxed," subtracting the exempt amount directly is a completely reasonable fix. Four independent runs converging on it is evidence that the intended rule was not inferable from what they were given. It is not evidence that effort failed.

So the fair score is 7 of 7 for every level. Flat again.

What effort did change here was, once more, output tokens: 3.6k, 3.6k, 5k, 10k. Wall time and cost followed. And scope creep showed up at low, high and xhigh, each of which invented a fuzzy gift-card category matcher that handles "Gift Card", "gift_card" and "GIFT-CARD" when the original code used an exact string. Medium alone made the exact minimal change the prompt asked for.

What I take from this

Effort controls how much the model deliberates, not what it knows. On every task here the answer was fully determined and reachable in one careful pass. Given that, low found it. Max found it too, after thinking about it for eleven minutes.

Verification loops make effort irrelevant. If the session can run tests, any level will iterate until green. The Python and Go benchmarks measured persistence, and persistence was free at every level.

Higher effort spends its budget on self-checks and defensive code. Oracles, throwaway verification scripts, extra validation, regexes that reject malformed input the spec never mentioned. Sometimes that is exactly what you want. In a codebase with a scope rule, it is unrequested behavior you now have to review.

Medium was the quiet winner on discipline. It never won on speed or cost, but on the bug hunt it was the only level that did precisely what the prompt asked and nothing more.

The total bill for this experiment was about $20. Roughly $9 for the Python runs, $7 for the Go runs, $4 for the bug hunt. The max Python run alone was almost a quarter of it.

Where effort probably should matter

I have not proven that effort is useless. I have shown three tasks where it did not help, and I think I know why. Effort should matter when deliberation itself is the bottleneck:

• Two-step inference. The symptom is in one file and the cause is in another, and the code near the symptom looks fine. A fix at the symptom passes the reported case and fails hidden siblings.
• A decoy fix that breaks a hidden invariant. An inventory patch that resolves the report and introduces a race under go test -race.
• A constraint hidden in a README that conflicts with the obvious fix.
• A codebase too large to read. Around 2,000 lines, where effort shows in search strategy and in what gets verified before editing.
• Repeat runs. Three samples per level at minimum. My single-sample wall times have variance comparable to the gaps between adjacent levels.

That is the next benchmark. Until then, my working rule: for a well-specified implementation task where the session can run tests, low or medium. Reserve high and above for tasks where the model cannot check itself and the answer is not sitting in the docstring.

And watch the clock. Nothing on max was wrong. It just took nine times longer to be right.

Comments

[vadim albarov]

Can anyone suggest a real‑world scenario where the effort level truly matters?