DEV Community: AI Dive

Your Claude Code subagents never touch Haiku 5.5. I measured what routing them there gives back.

AI Dive — Mon, 12 Oct 2026 10:29:03 +0000

Haiku 5.5 shipped on October 7 as "the subagent model", a tenth of the old price per token. I opened a month of my own Claude Code logs the same evening. 2,631 subagent runs. 42 on Haiku.

Not because I chose Opus for them. Because every subagent, the built-in Explore agent included, inherits the main session's model. Nothing goes to Haiku unless a file says so.

So I flipped them and asked the only question that matters on a plan: what does the usage meter actually give back when the workers run on Haiku 5.5?

What I actually tested

One real repo, tinydb, 226 tests in its suite. Seven tasks a subagent gets every day: explore the code, fix a failing test, clean lint errors, write a commit message, add a small feature, review a diff, summarize the architecture.

Four workers: Haiku 5.5 at medium effort, Haiku 5.5 at low effort, Sonnet 5.5, Opus 5.5. 182 runs in all. Every run was accepted or rejected by a hidden test or a fixed rule, never by eye.

Every headless run with streaming output emits one rate limit event carrying the five-hour and the weekly fraction, two decimals each, straight from Anthropic's response headers. I ran the four workers as four blocks, nothing else touching the account, and read the meter before and after each block.

Haiku passed everything and cost a fortieth of Opus

70 of 70 at medium effort. 70 of 70 at low. Sonnet and Opus passed their 21 each. On these seven tasks the quality gap did not show up.

The list price did. The small feature cost half a cent on Haiku low and 17 cents on Opus. Exploring the codebase: a third of a cent on Haiku, 14.5 cents on Opus. About forty times.

And Haiku was never slower. The feature took 17 seconds on Haiku low and 30 on Opus.

Anthropic's own caveat still stands: Sonnet and Opus remain better for complex agentic coding, and Terminal Bench puts Haiku at 39.2 against Sonnet's 70.6. None of my seven tasks is that kind of work. That is the point of routing: send the scoped tasks, keep the hard ones.

The setting everyone reaches for does nothing on its own

Since release 2.1.198 the built-in Explore agent inherits the main model, capped at Opus. Before that it ran on Haiku. The docs give the resolution order: the model named at launch, the model in the agent file, the subagent model variable, then the main model. And they say plainly that the variable alone does not change what Explore and Plan run on.

I checked with headless runs. Main session on Fable, one question: use the Explore agent to find where the query cache is cleared. The tool result names the model that ran. Opus 5.5. Same run with CLAUDE_CODE_SUBAGENT_MODEL=claude-haiku-5-5 and nothing else: Opus 5.5 again.

Two files fix it. The first is an agent file in your project:

.claude/agents/Explore.md

---
name: Explore
description: "Read-only codebase explorer. Use it to find where something is defined or called and to summarize what it found, with file paths and function names."
model: claude-haiku-5-5
effort: low
tools: Read, Glob, Grep, Bash
---

Claude Code treats a project agent named Explore as the Explore agent from then on. Same headless run: the tool result now reads Haiku 5.5.

The second is your settings:

{
  "env": {
    "CLAUDE_CODE_SUBAGENT_MODEL": "claude-haiku-5-5",
    "CLAUDE_CODE_SUBAGENT_MODEL_FORCE": "1"
  }
}

Force is the part people miss. Since 2.1.257 it applies the subagent model to every subagent, ignoring the overrides at launch or in an agent file. Which file for whom: the env block when every subagent should be cheap, knowing it also drags your deliberately Opus agents down. The agent file when only the explorer should.

Pin the full model id. The haiku alias means Haiku 5.5 on the Anthropic API only. On Bedrock, Vertex and Foundry it still means Haiku 4.5. And check: /tasks shows the model each subagent ran on. Trust that line, never the file.

The meter moved less than a point per block

Before and after each block:

Haiku 5.5 low,    70 runs: 5h window 21% -> 21%
Haiku 5.5 medium, 70 runs: 5h window 21% -> 21%
Sonnet 5.5,       21 runs: 5h window 21% -> 22%
Opus 5.5,         21 runs: 5h window 22% -> 23%
Weekly window, all 182 runs: 67%

The whole matrix cost $4.09 at list price. On the meter: two points of a five-hour window, zero of the week. The Haiku half cost 54 cents and moved nothing. One point of this account's window is worth more than 140 of these tasks on Haiku.

Anthropic publishes one sentence on model weight: Opus costs several times more per turn than Sonnet, and Sonnet more than Haiku. Several. No number. The windows are one pool for every model, so switching mid-session restores nothing.

The /usage breakdown is computed on your machine

The usage panel shows percentages per skill, subagent, plugin and MCP server. It looks like server data. The docs say it is approximate and computed from local session history. So the formula lives in the binary. Run strings on it and two functions fall out: a model tier (Fable 10, Opus 5, Haiku 1, anything else 3) and a record weight (cached tokens 1, fresh input 10, cache writes 12.5, output 50).

Run that on my blocks: 70 Haiku medium runs score 29.5 units, 21 Opus runs score 27.3. The panel would rank the Haiku block above the Opus block. The real meter said zero against one. Use the panel for what eats your local history, not for what a model weighs in your quota.

Where the week really goes

Andrea Grandi published his setup the day Haiku 5.5 shipped, on a Pro plan: Opus main session, a Sonnet implementer, and two Haiku agents, a scoper in plan mode and a shipper at low effort. 13 pull requests over 10 sessions, about 6% of the five-hour window per PR, about 1% of the week. His own line is the point: priced at Opus rates, those Haiku tokens would make the sessions only 5% more expensive. The saving came from the main context staying small: 75 requests became 54.

My logs say the same from the other side. Subagents took 48.1% of the tokens, 96% of everything was cache reads, and every worker starts with a launch prompt whose median is 47,000 tokens on this account, the same stack whether the worker is Opus or Haiku. The model switch does not shrink it.

Effort does change appetite. The same feature read 233,000 tokens over 14 turns at medium and 140,000 over 8 turns at low, for the same accepted result.

What this doesn't prove

One small repo. Seven scoped tasks under a minute each, not a day of agentic coding. A meter with one point of resolution, plus or minus a point per block, so Sonnet's point and Opus's point cannot be told apart. Grandi's dollars are an estimate on a Pro plan. And there is an open issue reporting 1,149 subagents out of 1,371 running on the parent's Opus model despite the file; on the current build every one of my 182 runs carried the configured model, but check yours.

The full run

The video shows the headless tool results, the meter reads block by block, and the strings output of the binary on screen.

When you route a subagent to Haiku 5.5, what do you send it? A) everything scoped B) only Explore C) nothing, the hard stuff is all I launch. I would like to know where the line sits for other people.

An AI assistant helped tidy the prose. The runs, the numbers and the opinions are mine.

The September cut took 17% of my Claude Code week. Subagents were taking 48%.

AI Dive — Thu, 08 Oct 2026 16:37:20 +0000

My Claude Code week started ending on Wednesday. I blamed the September 14 cut, like everyone on r/ClaudeAI did. Then I opened the usage screen and the biggest line was not me typing. It was the helpers I had told it to launch.

So I read a month of my own logs before touching a single setting. 455 sessions, 63,000 requests, and a number next to every saving tip I had been told to apply.

Which of the five fixes everyone lists gives a measurable share of the week back, and which one costs quality?

What I actually counted

Claude Code writes one JSONL file per session under ~/.claude/projects/, and one file per subagent run next to it. Every assistant line carries the usage of its request: input, output, cache reads, cache writes.

The first trap is double counting. The log writes the same answer about twice (1.96 lines per request in my month), so summing every line gave 18.6 billion tokens. De-duplicated on message id and request id, it was 9.3 billion. Every number below is the de-duplicated one.

The second limit is bigger. Anthropic publishes the weekly limit as percentages and plan multipliers, never as a token count. So everything here is tokens from one workload, not a share of your week. The order may change for you. The usage screen will tell you.

The cut itself is simple arithmetic. Call the old weekly limit 100. The summer promotion made it 150. The permanent level since September 14 is 125. From 150 to 125 is the 17% people feel, and Anthropic's "+25%" is true at the same time.

Subagents took 48% of everything

2,631 subagent runs in the month. They took 48.1% of all tokens and wrote 64% of all output, which is the heaviest line on the meter.

Each one pays an entry price. Before a subagent does anything, its opening request already carries a median of 47,117 tokens: the instructions, the tool list, the skill list, all sent again. Someone else measured 16 to 21k on a different machine with tiny agent prompts, and their line stuck with me: your agent file is a rounding error inside its own launch cost.

The model is the other half. A subagent inherits the main session's model unless the agent file says otherwise. Switch to the biggest model for yourself and every helper runs on it. In my logs the smallest model handled under 1% of subagent requests.

---
name: test-runner
model: haiku
---

Two habits, then. Skip the subagent for a job you could do in place. Pin a small model on the ones you keep. Nobody has measured what pinning saves as a share of the week, and a small model that needs more turns can cost more, so this stays a habit with a direction rather than a percentage.

The five-minute cache nobody lists

The main session's prompt cache lives one hour on a subscription. A subagent's lives five minutes. My logs agree with the docs on this: every cache write from a subagent landed on the 5-minute tier, every write from a main session on the 1-hour tier.

One setting moves it:

{ "subagentPromptCacheTtl": "1h" }

A Reddit user whose subagents wait on long builds saw cache writes go from 12 million tokens to 3 million after the change. In my own logs it barely matters: 2 requests in 1,000 arrived after a wait of five minutes, though each one rewrote about 75,000 tokens. If your subagents idle, turn it on. If they run in short bursts, leave it, because an hour of cache costs more to write.

One lunch break rewrote 130,000 tokens

This is the one I did not expect. A message sent within five minutes of the previous one wrote about 1,200 tokens of cache. After a pause of more than an hour, the median was 130,332. The session at that moment held about 175,000 tokens, so most of it was written again.

gap before the message   n        cache written (median)
< 5 min                  18,029   1,176
5 to 60 min              414      1,327
> 60 min                 79       130,332

Two cheap habits. Clear the session when a task is done, while the cache is still warm, so the next task starts small. And when Claude Code offers to resume a big session from a summary after a break, take it. A model switch mid-session empties the cache too, since each model keeps its own.

The caveat: 79 cold returns is a small sample, and some of them follow a compaction rather than pure idle.

Effort is the fix that can cost you quality

One developer ran the same 29 real tasks at all five effort levels. Mean cost per task went from $2.50 at low to $8.84 at max. Medium passed 28 of 29, more than any level above it. His words: the curve appears to peak at medium.

Then the catch. On the hard problems he picked, low passed 0 of 5 and high passed 5 of 5. A low attempt took two minutes, a high one thirty-three. So medium to build, high when a mistake is expensive, max almost never. Those costs are in dollars on an older model, not a share of your week.

The popular three weigh less than advertised

Removing MCP servers is the tip everybody repeats. Tool definitions are deferred by default now, and one article counted 1,350 tokens for 51 tools across three servers. A single server with one tool cost 18.

Turning off prompt suggestions made the rounds with a claim of up to 10% of the week. That was one account with enormous contexts. Another user on the same thread measured 3 to 4%.

Filtering shell output: the developer who measured it on his own usage found about a tenth of one percent.

All three grow with the size of your context. Switch them off if you like. Don't expect the week back.

What this doesn't prove

It is one month of one person's work, with heavy subagent fan-out. A solo session workflow will see a different split. The effort numbers are someone else's, on an older model, in dollars. And nothing here converts to a percent of the weekly cap, because the cap is not published in tokens.

The full run

The video shows the ranked table and the method on screen, including the counting trap.

What does your usage screen blame? A) subagents B) long sessions C) something I did not count. I would like to know whether 48% is my workload or the norm.

I used an AI assistant to tidy the prose. The logs, the counting and the opinions are mine.

I wrote a safety mod that crashes on purpose. Claude Code ran the command anyway.

AI Dive — Mon, 05 Oct 2026 16:37:28 +0000

I wrote the dumbest safety mod I could think of. It watches every shell command Claude Code is about to run, and it throws.

Then I asked for one command:

touch ./marker-failopen.txt

The guard crashed, as designed. One line showed up in the log: failguard: tool.call hook skipped: threw Error: guard crashed. And the file was sitting in the folder.

So what does a guard mod protect you from when the guard itself breaks? By default, nothing. That was the part of a week of testing that stuck with me most, and it is one of the reasons I deleted seven of the ten mods everyone told me to install.

What I actually tested

Mods shipped in Claude Code at the start of October: TypeScript functions inside a plugin that hook into an event and run before it, after it, or instead of it. Within a day, three tours were out telling people to install ten of them. None of them measured anything.

I took those ten and ran them against my own week: 85 sessions across 4 projects, 882 prompts, 5,993 tool calls. Each mod went through claude plugin validate, then ran the same task against a clean baseline on Claude Code 2.1.288, same machine, medians of three runs.

Six of the ten cost nothing I could measure at runtime. That does not make all six worth keeping.

A guard that throws is skipped

This is documented, not a bug. The events page says that when a hook with no .catch handler throws, times out, or returns a result of the wrong shape before calling next, "Claude Code skips it, and the next handler runs in its place."

It makes sense for decoration. A broken heatmap should not brick your session. For a guard it means the failure is silent and the command goes through.

The fix is in the same docs, and it is one handler:

on('tool.call', { tool: 'Bash' }, guard).catch(async ($, e, next) => { return { deny: 'The command guard failed, so this command was not run: ' + next.error.kind } })

Same crashing guard, same prompt, plus that catch: no file, and the model reports the block. One line decides whether your guard fails open or fails closed. If you installed a safety mod this week, open it and look for that catch.

A community re-test on 2.1.288 went further, 10 cases with 3 runs each, judged by marker files and not by what the model said. The catch pattern held. One path is still open: a deny returned after the call was already forwarded does not stop the tool. The file landed 3 times out of 3 while the model was told the write failed.

The most demoed mod bills you on every answer

next-steps is the one every video opens with. Your answer finishes, three suggested prompts appear above the composer, you press a number.

To get those suggestions it forks the session after each turn. Its README says so plainly: the fork shares the prompt cache, "so it costs about one short reply." On my bench that came to about 250 extra output tokens and 2.9 seconds per answer.

config	duration	output tokens
baseline	3980 ms	247
next-steps	6830 ms	497
cache-keeper	5569 ms	367

The detail I did not expect: the fork has no surface gate. Suggestions only draw in a terminal, but I watched the fork fire in a headless claude -p run where nothing can be drawn. There is no setting that keeps the suggestions and drops the fork. You can raise the answer length threshold or disable the plugin.

cache-keeper has the same shape, about 1.6 seconds per turn, plus paid pings to keep a cache warm that already lasts an hour on a subscription plan.

A bookmark that can start processes

claude plugin validate prints what a mod hooks, what it calls and which environment variables it reads. It takes a couple of minutes and it runs before you enable anything.

session-bookmarks was one of my six free mods at runtime. Its audit lists $.model.complete, $.process.run and $.fs.write. A bookmark that can call the model, run programs and write files. Nothing in it is malicious. It is just a lot of reach for the job, and I would not have known without reading two lines of output.

This matters because of what Anthropic said on launch day: "Mods run with the same access to your machine as Claude Code itself. They aren't sandboxed." Your permission rules cover Claude's tool calls, not the mod's own. The docs put it in one sentence: "with Read(.env) denied, a mod can still read that file with $.fs.read or start a program that does."

When a plain hook is still the right tool

Claude Code already had hooks, a shell script that fires on the same events. The measurable difference is the process spawn on every call:

hook body	per call	over 5,993 calls
bash -c 'exit 0'	8.3 ms	50 s
python3 -c 'pass'	26.1 ms	156 s
node -e ''	43.1 ms	258 s

A mod runs in-process and pays none of that. But four minutes a week only shows up at thousands of calls. My rule now: interface or event rewriting is a mod. Block, allow or log with a script you have read is a hook. A hook you have read beats a mod you have not.

What I kept

Three out of ten.

collision-guard. Two chats editing the same file is a real problem, and when it trips it asks in a dialog. It does not silently allow.
model-router. I ran a session on Sonnet, routed subagents to Haiku, and the usage readout showed the subagent billed on Haiku. About a third of the price for that job.
auto-handoff. It added 71 ms on the bench and does nothing until a context threshold, then writes the handoff once.

What this doesn't prove

One week, one machine, my workload, three runs per point on the small model. Two rows in my table moved in ways I put down to answer variance, so I only quote the deltas that held. Your three keepers might differ. The method transfers better than the list.

The full run

The video shows the crashing guard and the fixed one side by side, and the full ten-row verdict table with the measured cost of each mod.

Which mods are still on your machine after the first week? And did any of your guards have the catch handler when you checked?

I use AI tools to help tighten my drafts. The runs, the numbers and the opinions are mine.

Jev did not make Claude Code cheaper. Its own benchmark says so.

AI Dive — Sun, 27 Sep 2026 08:47:53 +0000

Six videos in five days wired Jev into Claude Code and called it the cheapest agentic coding loop available. I installed the gateway too. Then I did the thing none of those videos did, which is open the code.

One ternary in src/adapters/messages.ts, line 99, answers the entire question:

steer: thinking || cached ? "hint" : "tool_choice"

If thinking is on, or the conversation is cached, the gateway is only allowed to hint. Every real Claude Code request carries both. So hint mode is not a degraded fallback you hit occasionally. Inside Claude Code it is the only mode there is.

So what does a hint actually buy you?

What I actually tested

Jev is TypeSafe's decision model. No text generation, just a choice, a score or a yes-no probability, in 70 to 500 ms, at $0.042 per million input tokens. jev-gateway sits in front of Claude Code, looks at your tool roster, and tells the model which tool it thinks you want.

I did three things: read the gateway's adapter and launcher, read the benchmark the gateway author published alongside it, and put a logging proxy in the gateway's seat in front of my own Claude Code to count what a router would actually receive.

The install is honest, by the way. bin/clients.mjs sets ANTHROPIC_BASE_URL and nothing else, so your Pro or Max login keeps working and no API key is involved. The gateway's own README says it plainly: "Expect better tool picks on large tool lists, not lower cost or latency." Nobody quoted that line either.

The hint is two sentences, and the model can ignore them

In hint mode the gateway appends a block after everything the client sent, on the client's own blocks, so the cached prefix stays byte identical. The text is a <system-reminder> saying a tool-routing model suggests tool X. The model is free to disagree, and the wording says so.

It has to work this way. The Anthropic docs close the other doors:

Forcing a tool with extended thinking on returns an error, and on the newest models a forced tool returns a 400 regardless of thinking.
Changing tool_choice invalidates the messages cache.
Editing a tool definition invalidates everything.

The gateway touches neither, which is the correct call. It also means there is a guard that skips the hint entirely when the last block is a system reminder, and Claude Code appends those itself.

The benchmark nobody quotes is the gateway author's own

jev-gateway-bench is 120 sessions, 5 runs per cell, Claude Code 2.1, on the same subscription you use. Here is the part that never made it into a thumbnail.

On chess-bugfix, routing pays. Sonnet 5 read 48% fewer input tokens and finished 25% faster. Nothing got less correct.

On chess-san, adding a feature, it goes the other way:

Opus 5, routing on:  +61% input tokens  +47% requests  +83% time
Sonnet 5, routing on: +16% input tokens                +37% time

The authors' own explanation is the sentence I keep coming back to. A hint that does not fit costs a detour, instead of being ignored for free. A forced tool call you did not want is one wasted call. A suggestion you half follow is a path.

There is a correctness line in there too. One model solved the feature task five times out of five on its own and three out of five with routing on, using half the input tokens to get there. Cheaper and wrong is not a saving.

And the split that explains the whole product: Jev decided 76 to 100% of Codex requests, where the gateway forces the tool, against 34 to 51% of Claude Code requests, where it can only hint.

What your session hands the router before it decides anything

A router has to see your tools to pick one. So I counted mine.

clean install:  24 tools   47,000 prefix tokens
my real setup:  40 tools   57,000 prefix tokens
answer:         4 tokens

The gateway re-sends that roster on every call, and there is an open issue about exactly that. The bench repo has a footnote with a worse case than mine: one setup sent 285 tools and about 200,000 tokens with every Claude Code request, against 6 tools and 7,000 tokens clean.

Jev's own pricing makes this survivable. That is not the point. The point is that a decision model priced at four cents per million tokens is being handed a 47,000 token question to return four tokens, every turn, so that a model with thinking on can read two sentences it is allowed to ignore.

What this doesn't prove

It is one benchmark, two task families, five runs per cell, on a chess engine. My proxy numbers are my machine and my plugins, not yours. If your roster is small and your tasks look like bug hunts, the bugfix column is real and you should go measure it yourself, which is what the bench repo is for.

It also says nothing about the rest of the Jev surface. Compaction is a separate tool with separate tradeoffs, and Codex is where forced routing actually lands.

Verdict per use

Tool routing inside Claude Code: no. Hint mode is the only mode, and the vendor's own feature-task numbers go the wrong way.
fast-jev-compaction: not yet. 6,400 stars, and open issues on hooks not registering, full transcripts going to a third-party API, and dropped evidence producing fabricated work reports. The terms-of-service question is the one practitioners raised first.
Codex: yes. The gateway forces the tool there and steered up to every request.

Jev itself is cheap and fast. The detour around it is not.

The full run

The video walks the adapter line by line and puts the bench table on screen next to the claims made about it, which is easier to follow than a code block in an article.

Have you measured your own tool roster? If you run Claude Code with more than a handful of MCP servers, I would like to know what number you get, because mine surprised me.

I use AI tools to help tighten my drafts. The runs, the numbers and the opinions are mine.

Your agent commits every spec. Nobody owns them.

AI Dive — Mon, 14 Sep 2026 16:57:11 +0000

In June, the maintainers of okama, a Python finance library, found their internal implementation plans published on Read the Docs. Eight markdown files an agent had written, committed exactly as the plugin intends, rendered as public web pages. They noticed after the release.

The plugin was Superpowers. By default every spec and every plan it writes lands in docs/superpowers/, and the brainstorming skill commits the design doc before you've even seen the plan.

So who owns those files, and when is one dead?

Every tool commits the files

This isn't a Superpowers quirk. spec-kit keeps one numbered folder per feature on that feature's branch. Kiro keeps .kiro/specs/, one folder per feature. Anthropic's playbook goes further and commits an artifact at every stage: intent, spec, plan, diff, review findings, incident record.

Whether to commit is settled. Ownership and lifecycle aren't.

Picture a front-end team of 10 people in four squads sharing one repo, with every developer generating these files. Three failures are already documented:

A docs generator renders every markdown file under docs/, listed in the table of contents or not. That's the okama leak.
Issue #1246: the brainstorm commits the plan straight to main. One user counts 10 to 15 commits per session.
Drift, the quiet one. Birgitta Böckeler at Thoughtworks separates spec-first (written, used once) from spec-anchored (maintained for the feature's life). Superpowers writes spec-first documents and keeps them forever, so you get anchored storage with first-draft maintenance. Nobody updates the file, and an agent reads it next quarter as truth.

Rule 1: the folder has an owner, and it's a squad

Since v5, Superpowers honors the instructions in your CLAUDE.md over its own defaults. A table of paths isn't enough though. In issue #939 a user wrote an output paths table and the model followed the concrete path in the skill instead. One bold imperative line, naming the default as the thing to avoid, fixed it on the first attempt:

IMPORTANT: specs MUST be saved under docs/superpowers/<squad>/specs/
and plans under docs/superpowers/<squad>/plans/, NOT docs/superpowers/specs/.

In a monorepo, put that line in the package's own CLAUDE.md. Claude Code loads a subdirectory's file when it reads there. Then let the platform enforce the owner:

# CODEOWNERS
docs/superpowers/checkout/        @org/squad-checkout
docs/superpowers/search/          @org/squad-search
docs/superpowers/design-system/   @org/squad-design-system

Every spec review now lands with the people who'll live with it. One caveat: the override is a prompt, not a setting. Check the first spec after every plugin release.

Rule 2: not every plan is a team artifact

Personal redirects go in CLAUDE.local.md, which loads after the shared file and which you git-ignore yourself. For the folder itself, .git/info/exclude ignores it per clone without touching the shared tree.

The shared side is whatever passed the review gate the brainstorm already runs. Approved by a second person, it moves into the squad folder. Not approved, it stays local. Both are written on a branch, never on main. Two lines in the shared CLAUDE.md do it:

Treat specs and plans as the start of a feature.
Create the worktree before writing them.

There's a limit. The okama maintainers tried ignoring the folder first and reverted, because plans stopped syncing between machines and agents. Personal means personal.

Rule 3: a spec has a status, and a dead spec says so

Michael Nygard wrote this down for architecture decision records in 2011. A decision is proposed, then accepted, and when a later record changes it the old one is marked superseded instead of deleted.

So every shared spec opens with a short header, plus a review date if your team uses one:

---
status: accepted
superseded-by:
owner: @org/squad-checkout
touches: src/checkout/retry.ts, src/checkout/api.ts
---

Never edit an accepted spec into a different decision. Write the next one and point it back. The one exception is in Anthropic's playbook: when implementation departs from the plan, update the plan in the same commit.

The weak spot is that a status is metadata a human sets. The agent won't mark its own spec superseded unless your CLAUDE.md tells it to.

Rule 4: a spec that lies about the tree fails the build

For every spec whose header says accepted, pull the paths it names (in backticks or on the touches line) and check that each one exists. Proposed and superseded specs are skipped, because only accepted specs make claims about the tree.

#!/usr/bin/env bash
# drift gate: accepted specs may only name paths that exist
shopt -s globstar
fail=0
for f in docs/superpowers/**/specs/*.md; do
  grep -q '^status: accepted' "$f" || continue
  paths=$( { grep -oE '`[^` ]+/[^` ]+`' "$f" | tr -d '`'
             grep -m1 '^touches:' "$f" | sed 's/^touches://' | tr ',' '\n'; } | tr -d ' ')
  for p in $paths; do
    if [ ! -e "$p" ]; then
      echo "drift: $f names $p, which no longer exists"
      fail=1
    fi
  done
done
exit $fail

People already do the manual version. One commenter on r/ClaudeCode has Claude compare the spec files to the codebase and file tickets for what's missing. This does that on every pull request.

While you're in CI, add the guard okama shipped: exclude the folder from the docs build. In Sphinx that's one entry in exclude_patterns, and the files stay in git without reaching the HTML.

The check catches deleted files, not changed behavior. A spec can name files that all exist and still describe an API that's gone. That part is still review's job.

Rule 5: most work doesn't deserve a spec

Marmelab ran spec-kit on a feature that shows the current date and got 8 files and 1,300 lines of specification. The OpenSpec bake-off ran the same requirements twice, with the spec tool and with Claude Code alone. The spec run produced 50% more code with 50% more cyclomatic complexity, took twice as long and cost three times as much. It's one experiment, and the author says so. It also found three gaps the plain run missed.

A team that ran spec-driven development for months couldn't prove it improved the code: two to three times the tokens, about twice as long.

My rule: write a spec when the work crosses a squad boundary or will be read again in 90 days, and keep it near 300 lines. Everything else is a prompt. The spec buys coverage, not speed.

What this doesn't fix

None of these rules is enforced by a tool today. The override issue is open, the main branch issue is open, and the pull request that reordered the instruction was closed without merging. And if your team won't review a code owner request, it won't review a spec either. In that case git-ignoring the folder is the honest choice.

The full walkthrough

The video puts the issues, the ADR source and the okama config on screen. The written version with every source is at https://aidive.dev/videos/spec-plan-governance/.

Do you commit your agent's specs, and has a stale one misled an agent yet?

I used AI tools to help edit this piece.

Everything Claude Code costs 27,000 tokens before you type. I kept five pieces.

AI Dive — Tue, 08 Sep 2026 17:52:58 +0000

Someone asked on r/ClaudeCode whether everything-claude-code is really that good. The thread got one reply saying the token cost of skill descriptions is "minimal". Nobody had counted.

So I cloned it, version 2.2.1, and counted. It has 250,000 stars, 68 agents, 286 skills, 94 slash commands, 122 rule files and 23 hooks. The README that describes all of it is 2,200 lines long.

The question I wanted answered: what does a full install cost you before you type a word, and which parts are worth taking?

The part the repo tours skip

A skill's body only loads when the skill fires. Its name and description do not. They sit in your context from the first token of every session, and the same is true of every subagent.

That metadata is the price of admission, so I measured it on the fresh clone:

286 skill names + descriptions   85,000 chars   ~21,000 tokens
68 subagent descriptions         20,000 chars    ~5,000 tokens
the repo's own instruction file                  ~1,000 tokens
------------------------------------------------------------
standing context, full install                  ~27,000 tokens

Against a 200,000 token window that is 14% gone every session. It sounds survivable until you remember what shares that window: your files, your diffs, your tool results and the conversation itself. Every skill you will never call competes with the code you are working on.

The repo's own README says "optimize the context window, persist everything else". Then it ships 286 skills.

Two caveats before anyone quotes this. Tokens are estimated at four characters each, so the real number moves with the tokenizer. And none of this says the skills are bad. It says they are not free, and most of them are for stacks you don't write in.

What I kept

Five things. The first two don't need the repo installed at all.

1. AgentShield

A security auditor for agent configurations. It ships inside ECC and also lives on npm as its own package, ecc-agentshield (5,500 downloads last week).

npx ecc-agentshield scan --path .

It audits your agent files, hooks, MCP servers, permissions and secrets, with flags for output format, severity floor and safe auto-fixes. Its README justifies itself with numbers from January: 12% of one skill marketplace was malicious, 341 skills out of 2,800. Those are the package's claims, not mine.

The catch is a date. AgentShield was last published in March. The repo it ships with was updated yesterday, and the scanner's README still describes the ecosystem at 42,000 stars. Six months behind the thing it audits.

2. The settings block

The README publishes four changes to .claude/settings.json: default model to Sonnet, thinking budget from 32,000 down to 10,000, compaction threshold from 95% down to 50%, subagents on the cheapest model. It claims about 60% off model cost and 70% off hidden thinking cost. Its figures, not mine, and they are a trade. A smaller default model will cost you on the hard reviews.

3. Instincts

This is the one idea in the repo I have not seen anywhere else. An instinct is a small learned behaviour with a confidence score between 0.3 and 0.9. Not a rule you wrote. A pattern the system noticed you repeating and wrote down for you.

It watches before and after every tool call, runs the analysis in a background agent on the cheap model, and scopes instincts per project so your React habits stay in the React project. When the same instinct shows up in two projects it gets promoted to global. The system decides what is universal by watching instead of asking.

The tool behind it is 2,200 lines with six commands, and instinct libraries are exportable. Hand yours to a teammate and they import it, which is the first sane onboarding story I've seen for an agent setup.

String attached: all of it rides on the hook runtime, which is the exact part of the repo you are asked to opt into on purpose. No hooks, no instincts.

4. The three context skills

A context budget, a compaction adviser, a token budget adviser. The repo that costs you 14% also ships the tools to see it. The compaction rule alone is worth copying: compact after research and before you implement, after a milestone, after debugging, after an approach fails, and never in the middle of an implementation where you lose the file paths and the half-finished state.

5. The installer, used one skill at a time

Seven profiles, and the smallest says in its own description that it leaves the hook runtime out. You can skip profiles entirely and name skills one by one, and an adviser takes a plain sentence about your work and returns matching components with a preview before anything is written. Any install that would put hooks on your machine stops and waits for a yes.

Install that way and the bill stops being 14%. I'm not giving a figure for the fraction because I did not measure every combination. Even the smallest profile still carries the agents and the rules, so naming skills by hand is the only way to pay for exactly what you use.

What stays in the repo

The 68 agents as a set: about 5,000 tokens of standing context, mostly language reviewers and build fixers for C++, C#, Django and Kotlin. Take the two that match your work.

The 94 slash commands: the README itself calls them temporary entry points while the repo moves to skills, and commands are the easiest thing here to write yourself.

The hooks file: 23 hooks, 291 lines, 40 KB, every hook squeezed onto a single line, the longest at 2,000 characters. The descriptions beside them are readable and some are good (one blocks the agent from editing your linter config and says why). The description is the only part you can read.

The ballast: config for 19 harnesses and 15 MB of docs in 22 languages, for a tool you run in one language, in one place.

What this doesn't measure

Whether the skills make the answers better. I measured what they cost you to have. Those are different questions. Also, version 2.2.1 had 110 open pull requests the day I cloned it, so the counts above are already old.

Takeaways

A full install of ECC parks about 27,000 tokens in context before you type, 14% of a 200k window.
Skill bodies are lazy. Skill metadata is not.
The licence is MIT. Copying five directories out is the intended use, not a workaround.
AgentShield is worth running once, and worth checking the publish date of.
Instincts are the only new idea; they need the hook runtime.

Full write-up with the tables and chapter timestamps: https://aidive.dev/videos/ecc-what-to-steal/

Are you running the full install, a profile, or hand-picked skills? And what does your context sit at when a session starts?

I use AI tools to help edit my writing. The counts, the install and the opinions are mine.

JetBrains says rtk costs more. My dashboard says 11.6M saved. Both are right.

AI Dive — Sun, 06 Sep 2026 15:33:37 +0000

JetBrains spent about $320 of API credit in July to test rtk, the shell proxy tens of thousands of us installed to save tokens in Claude Code. Their sessions came out 7.6% more expensive per task. Two weeks earlier the same team measured caveman, the skill that advertises a 65% cut, and got 8.5%.

I read both posts with rtk gain open in another tab. It said 11.6 million tokens saved over 25,599 commands.

So who is wrong? Nobody. The two numbers count different things, and once you see which slice of the bill each tool touches, the four tools I run sort themselves into a ranking. That ranking is the useful part, so here it is.

What I actually tested

Four tools, all free, 575,612 GitHub stars between them on September 2: graphify, rtk, Superpowers and caveman. For each one I took the number it advertises, the number JetBrains measured in their paired benchmarks (claude-sonnet-5, 82 to 86 agentic tasks, run twice), and what my own machine shows after months of daily Claude Code use.

The frame that made it click comes from rtk's own docs. A Claude Code bill has two sides. Input tokens are everything the model reads: shell output, your prompt, the system prompt, and the whole conversation replayed on every call. Output tokens are everything it writes. JetBrains replayed 83 sessions and split the reading side:

1.9M characters of tool output, by origin
  19.7%  shell output rtk can compress
  46.3%  shell output rtk has no rule for
  34.0%  Read / Grep / Glob tools that never touch the shell

Only a fifth of what the model reads can pass through a shell proxy at all. Keep that table in mind for the rest.

rtk shrinks shell bytes, and shell bytes are not the bill

rtk sits between Claude Code and your shell. A PreToolUse hook rewrites git status into rtk git status, and the compact output comes back instead of the noisy one. My dashboard after 25,599 commands:

rtk find      354 calls   2.2M saved   46.6%
rtk read    3,504 calls   2.2M saved   10.4%
rtk grep    2,760 calls   1.9M saved   47.7%
rtk ps aux     24 calls   1.1M saved   98.0%
total      25,599 calls  11.6M saved   41.6%

That 11.6M is bytes divided by four, counted on the commands that had a rule. JetBrains found a rule for 349 of 1,056 shell commands, one in three, and combined with the table above that puts the ceiling around 3% of a session's cost. Their measured result was +7.6% per task at low reasoning effort (p=0.004), with 13.8% more turns, and no difference at high effort. Quality unchanged.

The rtk README now says it in one line: up to 90% of the bash output, which is not the same as cutting your bill by 90%. I still run it. One-line test output is nicer to read and it costs nothing. I just stopped expecting it to move the invoice.

caveman trims the smallest slice

caveman makes the agent talk like a caveman: drop articles, filler, pleasantries. Code, paths and error messages are never touched. Its own table shows 1,214 output tokens per answer without and 294 with, the 65% figure.

JetBrains ran it on 82 paired agentic tasks and got 8.5% of output tokens, 592k down to 542k, with no detectable quality change (sign test p=0.82). The reason is structural. An agent's output is mostly code and tool calls, which caveman leaves alone on purpose, so only the prose around them shrinks. Meanwhile the skill's rules cost about 1 to 1.5k input tokens on every turn.

Claude Code has shipped a built-in equivalent since v2.1.237, the Concise output style under /config. Same effect, no plugin. Both apply to the main conversation only; subagents keep their own prompt.

graphify moved the most tokens on a large repo

graphify parses the project with tree-sitter, locally, and builds a knowledge graph the agent queries instead of grepping files. Zero model calls for the code pass. I ran its built-in benchmark on one of my projects:

nodes                  34,031
edges                  56,865
naive full read     2,268,733 tokens
average graph query    24,702 tokens
                        91.8x fewer per query

Two caveats that the number hides. The comparison is against reading everything, and a grep-driven session was never that expensive, so the real gain is smaller. And the graph goes stale; you rebuild it with graphify update or the git hooks, and the semantic pass over docs and PDFs does spend tokens. On a small repo the gap shrinks to nothing.

Install: uv tool install graphifyy (double y), then graphify install.

Superpowers moved the bill, without compressing anything

Superpowers is Jesse Vincent's plugin, 280,792 stars, fourteen skills. Nothing in it compresses output. It changes what gets read and which model reads it.

A brainstorming gate blocks any code until the intent is approved. The plan skill cuts work into steps of two to five minutes: failing test, minimal code, tests green, commit. Each step then runs in a fresh subagent whose context holds only that task, never the session history. And the rule that pays for everything, quoted from the skill file: "Use the least powerful model that can handle each role to conserve cost."

mechanical, well specified   ->  small model
multi-file, debugging        ->  standard model
architecture, final review   ->  most capable model

That is what makes Opus and Fable usable on a $20 Pro plan. The expensive model touches a handful of tasks instead of the whole session. The cost is ceremony: the gate fires on a one-line fix too, and that is tokens as well.

What this doesn't prove

My graphify and rtk numbers come from my own projects, and repo shape changes them a lot. JetBrains measured on Sonnet 5 with 82 to 86 tasks; a different model at a different effort level can move the rtk result in either direction, as their own high-effort arm shows. I did not benchmark Superpowers with a paired test; its effect is on model selection, which is visible on the invoice but harder to isolate per task.

The full run

The video walks through the JetBrains tables and my dashboards on screen, with the four tools ranked at the end: Superpowers, graphify, rtk, caveman.

What do you run to save tokens, and did you ever check it against the bill? A) a shell proxy B) a code graph C) task and model discipline D) never measured, just installed

English isn't my first language; I used an editing tool to polish the wording. The measurements and opinions are mine.

Fable 5.1 cut cache reads by 75%. Cost per task went up 20% anyway.

AI Dive — Fri, 04 Sep 2026 15:34:54 +0000

In the same week that Claude Fable 5.1 shipped, two things happened that look like they cannot both be true.

Millennium had a piece of code that crashed about one run in a million. Their team had not explained it in four to five years. Every model they tried missed it, Fable 5 included. Fable 5.1 found it.

That same week a creator titled his review "I'm asking for a refund", and Artificial Analysis, which ranks these models for a living, put Fable 5.1 at the top of its intelligence index while measuring it at 20% more cost per task than Fable 5.

Both are accurate, and the reason is not in the benchmarks. It is in the pricing page and in how you pay for the model.

What I actually did

I did not run my own benchmark suite. There are plenty of those already and they mostly disagree.

What I did was read the release the way you would read a spec sheet before buying hardware: the announcement, the model docs, the system card, the prompting guide, the pricing page, and the support articles about plan limits. Then the independent measurements from Artificial Analysis, ARC Prize, FrontierSWE and Simon Willison, because the vendor's own table is not evidence about the vendor.

That turns out to be the differentiator. Almost every review this week ran demos and picked a side. The interesting material was sitting in the footnotes.

The cut is one line item

Fable 5.1 is not broadly cheaper. Input is still $10 per million tokens and output is still $50, unchanged from Fable 5.

One line moved. Cache reads, meaning the tokens the model rereads from a prefix it already processed, went from $1 to $0.25 per million. That is the entire price cut.

It matters more than it sounds, because in a long agent session most of your tokens are re-reads of the same context. Anthropic measured four weeks of real August traffic and reports roughly 25% off a typical bill and up to 45% off a heavily agentic one.

It also produces the one situation where the expensive model undercuts the cheaper one:

Cache read, per million tokens
Fable 5.1   $0.25
Opus 5      $0.50

Everything else on Fable costs double what Opus costs. A cache heavy loop really can be cheaper on Fable. Nothing else can.

Why the per-task number still went up

The second dial is effort, and it moves the bill much harder than the cache line does.

Simon Willison drew the same pelican at every effort level. Low cost him 10 cents. High cost 13 cents. Xhigh cost $1.83. Max cost $3.30, producing 65,927 output tokens over 14 minutes. Same prompt, 33 times the price.

Artificial Analysis runs everything at max. So they measured $3.76 per task against $3.14 for Fable 5, which is the 20% increase people are quoting. The cache cut is why it was not worse: without it the same measurement would have landed at $5.16.

So the honest version of the headline is that the model got cheaper to reread and more expensive to think, and which of those dominates depends on your workload. If you are running one shot requests at max effort, you are the case they measured. If you are running a review loop over the same repository for two hours, you are the case Anthropic measured.

The plan is a different meter entirely

None of that per-token math applies if you are on a subscription, and this is where the refund videos come from.

On Pro, Fable is not part of the plan's usage limits at all. It runs on pay as you go usage credits, and unlike previous launches there is no one-time credit this time.

On Max it is included, for up to 50% of your weekly limits. The same support page adds that these models consume those limits faster than other Claude models. That sentence is doing a lot of work and almost nobody quotes it.

Then the counter itself. The pricing page says Max gives 5x or 20x more usage per 5-hour session than Pro. Per session, with weekly limits stacked on top. A lawsuit filed June 15 alleges the real weekly multipliers are far below the advertised ones. And on September 14 a permanent 25% raise of weekly limits takes effect, which Anthropic itself clarified works out to a 17% reduction compared to today, because a temporary 50% boost ends the day before.

The model is not more expensive per token on a plan. It spends a budget faster than that budget was ever advertised to be spent.

Three things break if you call the API today

If you have a working Fable 5 integration, three changes fail it rather than degrade it.

Forced tool use returns a 400. Setting tool_choice to any or to a named tool is rejected. Thinking is always on now, and forcing a call would skip it.

Thinking blocks record which model wrote them, one way. Fable 5.1 reads Opus's thinking. Nothing reads Fable 5.1's.

Conversations are append only. Edit an earlier turn, or rebuild your system prompt or tools array between requests, and the next call fails with a message about the block being bound to a different conversation. This is enforced for accounts created on or after August 31. It breaks every harness that injects a reminder and deletes it on the following turn, which is a very common pattern.

Five things arrive at the same time, and the useful one is that you can change effort per message without losing your cache.

Then there are the documented behavior regressions, which are the ones burning people's quota right now. It rewrites whole files for small edits. It issues one tool call per turn in coding loops where Fable 5 batched several. It sometimes describes what it would do next instead of doing it. Each of those has a one line fix in the prompting guide, and each of them costs output tokens until you apply it.

What this doesn't prove

This is a documentation read, not a capability test. I cannot tell you whether the model is better at your work, and I did not try to.

The benchmark table is also weaker evidence than it looks. Anthropic's own footnote gives a standard error of 3.5 to 4.5 points per model, which puts the three point coding lead over Opus 5 inside the error bar. The rows missing from the marketing graphic are the ones where Opus 5 wins: 90.42 against 90.0 on ARC-AGI-2, and 59.4 against 54.7 on SWE-bench multimodal.

One more thing worth knowing before you send anything sensitive: Fable 5.1 carries 30-day retention and is not available under zero data retention unless Anthropic authorizes it. Enterprise Frontier Safeguards is the way out, and it ships this fall.

The full run

The video walks the actual pages, so you can see where each of these numbers lives rather than taking my word for the quote.

If you are on Max: did your 5-hour window survive the first week of Fable 5.1, and did you change your prompts before or after finding out?

English is not my first language, so I use a tool to polish the writing.

Superpowers fixes Claude Code. Then it bills you for every two-line fix.

AI Dive — Wed, 02 Sep 2026 16:40:40 +0000

Every few weeks the same thread comes back on r/ClaudeCode. Someone installed the Superpowers plugin because everybody recommended it, checked their usage stats, and found it sitting at 1 to 3 percent. No visible change in the code either. Is it worth it, or is it a tokens killer?

The plugin has 280,000 stars, 25,000 forks and is on its sixth major version in under a year, so this is not a prompt pack somebody abandoned after launch. Jesse Vincent, who has been shipping open source developer tools since the 90s, built it by mining 2,249 markdown files of lessons his own agents had learned.

I installed it, read the 14 skill files line by line, and ran it on my own projects. The short answer to the Reddit question: both camps are right, and which one you land in depends on what you feed it.

What I actually looked at

Superpowers is a free plugin on Anthropic's plugin marketplace, one command to install. The core is 14 skills: markdown instruction files the agent loads whenever a situation matches. Brainstorming, plan writing, subagent driven development, test driven development, systematic debugging, verification before completion, and so on.

A hook at session start forces Claude to check, before every task, whether one of these skills applies. The entry skill is blunt about it: if there is even a 1 percent chance a skill is relevant, load it. So this behaves less like a toolbox and more like a development methodology injected into your agent.

I read every file, then watched what each one did on real feature work and on real small fixes.

The gain sits upstream of the code

The brainstorming skill opens on a hard gate: no code, no scaffolding, no implementation skill of any kind until you have approved an explicit intent. Every request gets sorted into one of three paths. A spike is a feasibility question whose output is an answer, not code you keep. Bounded is a small change to a flow that already exists. Architectural is anything that restructures how the project fits together. The agent says its classification out loud so you can override it, and hidden complexity found mid task can upgrade the path, never downgrade it.

On my own project this phase has already killed features I would have built for nothing. That alone paid for the install.

Out of the architectural path come two documents: a spec you validate, then an implementation plan. The plan skill's opening instruction sets the tone. Write it for a skilled developer with zero context on the codebase and, in the file's own words, questionable taste. The work gets cut into tasks of two to five minutes each:

write the failing test
run it, confirm it fails
write the minimal code that passes
run the tests again
commit

Each task lists the exact files to touch, down to line numbers. This sizing is the reliability argument in one line: a short task means an agent that finishes with a context window still mostly empty. It never reaches the moment where the session overflows and the agent starts inventing functions that do not exist.

The cost is the execution loop, and it never switches off

At execution time your main session stops coding and becomes an orchestrator. It dispatches one fresh subagent per task, with exactly the context that task needs and none of your session history. When the subagent is done, the orchestrator runs a two part review: spec compliance first, then code quality. If the review fails, a fix loop starts, capped at five rounds. The first three resume the original implementer, round four dispatches a fresh one on a more capable model, and at round five a breaker trips and the orchestrator rules on every open finding itself.

That is a real code review process running on its own, task after task. It is also where the tokens go. Dispatch briefs, two reviews per task and the completion ledger are paid every single time, and you feel it on the smallest tasks.

The brainstorming file is explicit that the ceremony scales with the task. The human approval step does not. On a two line fix, that means answering framing questions, approving a two sentence design, then waiting for the full cycle. For a typo in a config file, the complete process is simply slower than fixing it yourself.

And here is the answer to the 1 to 3 percent question. If your usage stats show the plugin that low, your requests almost never trigger the skills. You are paying the session start check every time without ever touching the gains.

The setting that decides whether it pays

The execution skill has a model selection section that starts with one rule: use the least powerful model that can handle each role. A well specified mechanical task touching a file or two goes to a small model. Coordination across several files and debugging go to a standard model. Architecture and the final branch review go to the most capable one.

Two subtleties in that file I had not seen anywhere else.

First, always name the model explicitly when you dispatch. A subagent without one inherits your session's model, often the most expensive one, which silently defeats the whole section.

Second, turn count beats token price. The cheapest models take more turns on multi step work and end up costing more overall, so reviewers and implementers working from prose get a floor one tier up.

This is what makes running Opus or Fable on the 20 dollar Pro plan workable with the plugin: the expensive model only touches the few decisions that deserve it.

The win nobody installs it for

Specs and plans are not chat messages that vanish. They are markdown files saved in a dated plans folder inside the repo and committed with the work. The spec travels with the plan, and conflicts resolve against the spec, so the document is the authority rather than the agent's memory.

Six months later you mention the plan file in a prompt and the agent picks the feature's context right back up. Your git history stops telling you only what changed and starts telling you why. Documentation falls out of the process without anyone sitting down to write it.

What this does not prove

I read the files and used the plugin on my own projects. I did not run a controlled benchmark with token counts per task, so "it burns quota on small fixes" is an observation, not a measurement. The repo also moves fast: sixth major version in under a year, 125 open issues, so the skills you read today will have changed by the next update.

The plugin plans its own exit, though. Its instructions put your directives above the skills, so telling the agent to skip the process works.

The rule I ended up with

Superpowers on by default for any feature work. An explicit skip for tiny fixes.

If you build features that take hours with Claude Code, install it. If your usage is throwaway scripts and small fixes, walk on by. In between, install it and learn to say skip, or let just the brainstorming skill run for a few days. It carries most of the gain and the other skills graft on after it.

The full read

The video shows the actual skill files on screen, section by section, which is hard to convey in prose.

Which camp are you in? A) On all the time B) On for features, off for fixes C) Deleted it, saved my quota D) Never installed it

The reading, the runs and the opinions are mine. I used an editing tool to polish the English.

Anthropic graded 400,000 Claude Code sessions. Managers beat the engineers.

AI Dive — Mon, 31 Aug 2026 15:29:55 +0000

Anthropic ran a classifier over roughly 400,000 Claude Code sessions from 235,000 people, recorded between October 2025 and April 2026, to figure out who actually succeeds with a coding agent. The finding that made me read the whole thing: management occupations reach verified success slightly more often than software engineers, and the ten biggest occupation groups all land within seven points of the engineers.

I made a video that walks through the study and then tests its main claim live. Here is the short version.

What "verified success" means

The numbers only make sense with the definitions. Nobody at Anthropic read the transcripts. A classifier built on Sonnet 4.6 graded each session, and its grades were cross-checked against telemetry (commits, code changes, test results). On sessions that touch code, classifier and telemetry agree more than 90% of the time. A success only counts as verified when there is hard evidence, like passing tests or committed work.

The five levels

Each session also gets an expertise rating from novice to expert, based on how the person works, not on their job title. A lawyer who knows exactly what a contract script must catch can rate as an expert. A senior engineer poking at an unfamiliar language can rate as a novice.

Novice-rated sessions reach verified success 15% of the time. Intermediate through expert sit at 28 to 33%. Almost the entire gain is between novice and intermediate, which is the encouraging part: you do not need mastery, you need to stop prompting like a novice.

The output gap is just as stark. One instruction from a novice triggers about 5 actions and 600 words of output. The same instruction from an expert triggers about 12 actions and 3,200 words.

The part I tested myself

In the video I run the same task twice in two real Claude Code sessions, once written like the novice prompts in the study and once like the expert ones. The expert prompt contains zero code. It packs a repo map, constraints, and a test plan, and the session behaves almost exactly like the study predicts.

Three habits the data rewards, if you want the actionable bit: frame directions precisely up front, ask the agent to verify its own work, and redirect a failing session instead of watching it sink. When things break, 19% of novices quit with zero lines written; everyone else quits 5 to 7% of the time.

Caveats

Verified success caps out around a third even for experts. The data is self-selected Claude Code users. And the occupation ranking may partly reflect who confirms out loud when they got what they asked for. The study is still the best evidence I have seen that knowing your domain now matters more than knowing how to code.

Full write-up: https://aidive.dev/videos/anthropic-400k-expertise/