DEV Community: Ken Imoto

Claude Code vs Cursor vs Codex: 31 Days of Real Receipts, and the Cheapest Isn't Who You Think

Ken Imoto — Wed, 08 Jul 2026 13:00:01 +0000

Every "which coding agent is best" post I read ranks accuracy, plugin support, and vibes. Almost none of them show a monthly bill. This is the post I wanted to read before I picked one.

For the 31 days of June 2026 I ran all three official coding agents on the same laptop: Claude Code (Anthropic), Codex (OpenAI, inside ChatGPT), and Cursor. Same repos, same tasks, same me. I kept receipts. I logged sessions. I ran a local Qwen 3.5 35B on an RTX 4070 in parallel to see where owning silicon paid back.

Short answer: one of them is cheapest for me right now, but not by the margin the marketing pages suggest, and if my usage shape changes even a little the ranking flips. That is the real headline. The bill depends on what you do more than on which logo you pick.

What "one month" was, in detail

The month I logged:

22 working days at the keyboard.
Roughly 6 hours a day inside an agent, mixed reading + writing.
Two projects: one TypeScript SaaS with ~180 files, one Python data pipeline with ~60.
Three "hard" refactors (multi-file, week-scale). The rest was normal feature and bugfix work.

Usage shape matters more than the number on the pricing page. A part-time hobbyist and a full-time engineer buying the same $200 tier are buying two different products at the same price. The line I care about is not $/month, it is $/hour-of-agent-time.

The three cost shapes

Every coding agent bills you as one of three shapes, and June 2026 has not changed that.

Subscription with usage multiplier. Flat fee, soft cap on requests, throttle when you're over. Claude Code Pro / Max, ChatGPT Plus / Pro, Cursor Pro / Ultra all fit here.
Metered API. Per token. No monthly ceiling unless you set one.
Local GPU. You bought the card. Electricity + amortization. Zero variable cost per token but a fixed capacity ceiling.

The break-even between these three is what most posts skip. My June numbers, in one table:

Claude Code: $200/month Max, and I hit the throttle twice

I ran Claude Code on the Max 20x plan at $200/month, per Anthropic's current pricing page. The Pro tier at $20/month exists but is calibrated for a few focused sessions a day, not an agent-driven workflow.

What the month bought me:

Claude Code was my primary tool for the two hard refactors. Long-context is where it earns the bill.
I hit the 5-hour rolling limit twice on Max, both times during a deep multi-file refactor where I was running Sonnet 4.6 in the background across 3 subagents. That is Anthropic's cost tell: subagents multiply token spend, and the Max tier feels the cap when you fan out.
Everything else fit comfortably.

If I had run the same month on the metered API at Sonnet 4.6 pricing of $3/M input and $15/M output (the introductory $2/$10 promo runs through August 31, 2026, so I'm quoting the standard rate), the honest estimate for my token volume landed somewhere in the $260-$380 range. The subscription won by ~$60-$180, at the cost of a rate limit I can predict but not remove.

Specific lesson: for a heavy user, Max 20x beats the metered API only until you fan out subagents. Anthropic's own engineering blog notes that multi-agent runs use roughly 15x the tokens of a single-agent chat. That's real. If I ran three subagents on Sonnet as my normal shape, the metered path would probably beat Max.

Codex: bundled inside ChatGPT Pro at $100, and the metering just changed

Codex is the odd one. It does not have a standalone subscription. It rides inside your ChatGPT plan.

I ran ChatGPT Pro at $100/month, the tier OpenAI added in April 2026 to sit between the $20 Plus and the $200 Pro-20x. Two things the marketing page does not emphasize:

On April 2, 2026, Codex pricing moved from per-message to API-token-equivalent metering. If you were used to the old plan, your "same amount of work" started drawing down credits at a different rate. Nobody's monthly bill stayed the same, they just moved.
The average Codex developer sits at roughly $100-$200/month across all instances they're running. That is the same number I hit. Bundling with ChatGPT means I paid one bill, but the token math is not hidden. It is just billed as one line item.

Where Codex earned the seat for me: quick front-end scaffolding, one-shot script generation, "explain this stack trace" flows where I want an answer inside a browser tab I already have open. Where it did not: long-lived agent sessions in a real repo. That is where Claude Code held the line.

Cursor: $60/month Pro+, and I stopped forcing frontier models

Cursor's June 2025 credit model change means the plan you buy is really a credit pool. Hobby is $0, Pro is $20, Pro+ is $60, Ultra is $200. I paid for Pro+ this month.

The move Cursor pushed on me was Auto mode. On any paid plan Auto is unlimited: it picks a model for you and does not touch your credits. If you leave Auto on for tab completion and short edits and only reach for Claude Sonnet or GPT-4o for the actual reasoning work, the credit pool lasts.

I did the opposite for the first two weeks and paid for it. I forced Claude Sonnet everywhere, my credits drained by day 18, and I bought a top-up. The last two weeks I flipped the pattern: Auto for autocomplete, frontier models only when I explicitly asked. Same Pro+ pool covered the rest of the month with room to spare.

The Cursor-specific insight nobody puts on the pricing page: Pro at $20 is fine if you use Auto for most things. Pro+ at $60 is where you land if you cannot help yourself and keep clicking Sonnet. Ultra at $200 is for someone running agent mode all day; that was not me this month.

The local GPU: RTX 4070 + Qwen 3.5 35B, the breakeven that is not what you think

I keep an RTX 4070 running Qwen 3.5 35B locally, and I ran it as a fourth "agent" in parallel to see which jobs made sense to send there.

The naive breakeven math is easy and misleading. A $600 card amortized over 24 months is $25/month. Electricity at moderate use is another $10-$15. On paper: $35-$40/month for unlimited tokens.

Here is what the naive math misses:

The 4070 cannot run the frontier models. It runs the small ones, competently but at ~40% of the quality of Sonnet 4.6 on the tasks I care about. For me, that eliminated it from primary agent duty.
Where it did earn its slot: bulk classification, refactor precheck, and offline batch jobs. I ran a 3,000-file "which files touch payment logic" scan on Qwen overnight; the same job on Claude API would have been $8-$12. Twenty of those a month is where the card breaks even.
Zero of my long-context refactor sessions belonged on local. The context window on Qwen 3.5 35B is not comparable to Sonnet 4.6.

Local is not a subscription replacement. It is a batch-workload absorber. Buying a 4070 to save money on your primary agent is buying the wrong tool for the wrong job.

The 31-day scoreboard

Here is what landed on the credit card and what I got for it.

Tool	Plan	Monthly cost	Where it earned it	Where it did not
Claude Code	Max 20x	$200	Long refactors, subagent runs	Rate-limited twice on fan-out
Codex	ChatGPT Pro $100	$100	Quick scaffolding, browser-tab flow	Multi-file agent sessions
Cursor	Pro+	$60	Autocomplete + Auto mode	Credit drain if you force Sonnet
Local (Qwen 3.5 35B)	RTX 4070 amortized	~$35	Bulk scans, batch classification	Long context, frontier quality

Total this month: $395. Yes, all four. The overlap is not wasted; each tool covered a workload the others charged 3-5x more to do.

Now the honest bit. If I had to pick one and drop the rest:

If my month were 90% multi-file agent refactors and I could tolerate a rate limit, I would keep Claude Code Max 20x and drop the other three. Roughly $200/month.
If my month were 90% "one file at a time, browser tab open," I would keep ChatGPT Pro at $100 and use its Codex allocation. Roughly $100/month.
If my month were 90% autocomplete-first coding with a couple of hard problems a week, I would keep Cursor Pro at $20 with Auto default and pay-as-I-go for Sonnet. Roughly $20-$40/month.

The reason I keep all three is that none of my months look like "90% one shape." Yours probably do not either. But if you have to pick, pick against your dominant workload, not against the benchmark you saw on Twitter.

What I will drop for July

Two changes I'm making for the July run.

Cap the Claude Code subagent fanout at two. I hit the throttle twice this month, both times because I told it to run three subagents in parallel. Anthropic's own doc reports 15x token spend on multi-agent runs; I don't need three, two solves most of the problem, and I stay under the rate limit.

Stop reaching for Sonnet in Cursor. Pro at $20 with Auto default was clearly the right tier for me. The $40/month I saved on Cursor pays for something more useful than a habit.

If I write this post again at the end of July I'll compare the two months side by side. The Twitter version of "which agent is cheapest?" is the wrong question. The version worth answering: what is my usage shape, and which of these four bills matches it?

That answer changed for me twice this month. It will probably change again. But I'll have receipts.

The book I extracted the receipt-first playbook from

If you liked the receipt-first framing, I turned the general "estimate cost before shipping the prototype" mindset into a full playbook in Claude Code Mastery. It covers the subagent fanout math, when Skills earn their token cost, and how to keep a heavy month from turning into a surprise. Same posture as this post: real numbers, then decisions.

Sources

Claude Code / Anthropic API pricing: claude.com/pricing.
ChatGPT / Codex pricing: chatgpt.com/pricing and developers.openai.com/codex/pricing.
Cursor pricing: cursor.com/pricing.
Anthropic multi-agent token math: Anthropic Engineering — Multi-Agent Research System.
Simon Willison publishes ongoing coding-agent usage-shape breakdowns worth reading alongside: simonwillison.net.

The Compaction Plugin I Was Releasing Warned Me Mid-Release

Ken Imoto — Tue, 07 Jul 2026 13:00:00 +0000

At 3 a.m. I was finishing the release of compact-ops, a Claude Code plugin that protects sessions from context compaction. Context usage hit 67%, and a notification appeared in my own session: threshold crossed, run /compact at a clean stopping point, here is your current plan and your latest decision. I wrote that warning. I was also the person it saved.

Twenty minutes later the same session made the opposite point. I ran /compact, and all three of the plugin's hooks failed. During the release I had restructured the plugin's directory layout, so the running session was still holding a stale path. Compaction itself finished normally, because every hook is fail-open: if it breaks, it steps aside instead of blocking the built-in behavior. One session, one night, and I got to watch the feature work and watch it fail safely. You can't buy dogfooding like that.

This post covers what /compact actually throws away, what I borrowed from the plugin this one is derived from, and the six things I hardened before making the repo public.

What /compact actually throws away

When a Claude Code session fills its context window, the whole conversation gets compressed into a single built-in summary and the original messages are discarded. Manual /compact and auto-compact both work this way.

The summary is decent at code. What it drops is operational state: "the push was already approved", "we tried that approach and it failed", "that number came from this file". When the post-compaction agent forgets those, it re-asks for permissions you already granted and re-attempts fixes you already buried. That's the failure mode I wanted insurance against.

Here is the timeline with and without the plugin:

Moment	Standard Claude Code	With compact-ops
Usage passes 60% (configurable)	No warning; auto-compact arrives unannounced	One-shot reminder plus a 3-line recitation of plan / phase / latest decision
At compact time	Built-in summary only	Same compression, plus a transcript backup and a separate LLM writing a 10-heading state file
Right after compact	Agent continues from the summary alone	State file and a "re-read the originals first" note injected into the fresh context
After the session ends	Summary exists only inside that session	State persists for 30 days and is re-injected on `claude --resume`
If a hook fails	--	Fail-open; standard compaction proceeds untouched

The compaction algorithm itself is untouched. Everything runs through official hooks, from the outside.

Standing on compact-plus

The core idea is not mine. u-ichi's compact-plus (MIT) got there first: a PreCompact hook that calls a separate LLM to write structured state before the summary eats everything. It's the kind of tool you want installed the moment you read its README.

My setup had three mismatches with its assumptions, so I built a derivative instead of a fork and changed exactly three things:

The usage warning works with nothing but the plugin. compact-plus reads a marker written by a statusline script that lives in the author's separate dotfiles repo, so a plugin-only install never fires the warning. compact-ops computes usage inside a UserPromptSubmit hook, straight from the latest message.usage in the transcript.
State survives reboots. compact-plus keeps state in $TMPDIR, which is gone after a restart. If you resume yesterday's work every morning, that's no insurance at all. State now lives under ~/.claude/compact-ops/, organized per project, kept for 30 days.
Recovery also fires on --resume. Not just across a compact. A SessionStart hook injects the same state when you reopen the session the next day, reboot included.

I also made the LLM backend Claude-only (Sonnet primary, Haiku fallback). compact-plus falls back to Codex, which assumes a ChatGPT Pro subscription sitting next to your Claude one.

A state file with 10 fixed headings

Before each compaction, a separate LLM writes a markdown file that always has the same 10 headings: Active Plan, Current Phase, TaskList Summary, Session Decisions, Constraints and Blockers, Worker Topology, Skills Invoked, Editing Files, Failed Attempts, Recovery Notes. The interesting ones are Session Decisions and Failed Attempts -- precisely the operational facts that built-in summaries tend to thin out.

The warning side reuses the same file. When usage crosses the threshold, the injected reminder isn't just "compact soon" -- it recites the top of your current state so the agent keeps the big picture during the messy turns right before a compact:

[COMPACT WARNING] Context usage reached 67% (134,102 / 200,000 tokens).
State recitation:
- Active Plan: Ship compact-ops v0.2.0 (hardening + marketplace fix)
- Current Phase: post-release verification
- Recent Session Decision: circular symlink removed; source is "./" now
- At a natural work boundary, tell the user they can run /compact as-is.

That recitation is what appeared in my session at 3 a.m. Reading your own plan back, written by a hook you wrote, is a strange kind of code review.

So the post-compaction agent restarts with two anchors: the standard summary and the structured state file. But the state file is never treated as authoritative. The injected recovery guidance explicitly says to re-read the original project files before trusting anything. An LLM summary trusted blindly by another LLM is just a faster game of telephone.

It's also a plain markdown file. When a handoff feels off, you can open it and see exactly what the previous session thought it was passing along.

Six things I hardened before going public

v0.1.0 was "works on my machine". Before flipping the repo public I ran my own review plus a second pass with an independent CLI reviewer, and shipped v0.2.0 with six fixes:

Permissions. State files and transcript backups contain your raw conversation, including any secrets that tool output echoed. Every hook now sets umask 077; directories are 700, files are 600.
session_id validation. Hook input JSON was flowing straight into file paths. It now passes an allowlist (alphanumerics plus ._-, no ..) before touching the filesystem.
LLM output validation. The state generator used to be trusted after a first-line check. Now all 10 headings must be present, or the previous state is kept and the write is skipped.
Single-pass jq. The transcript squasher spawned jq three or four times per line, which on a long session could eat the hook's entire timeout budget. It's one streaming process now.
Gzip backups. Transcript JSONL compresses to roughly a tenth of its size.
Debug logging. The real lesson. Fail-open design doesn't get in your way in production, and it also dies in complete silence. The three hook failures at the top of this post were only visible because the harness printed an error line; the plugin itself had no way to tell me why. COMPACT_OPS_DEBUG=1 now logs every swallowed failure. If you're writing fail-open hooks, write the logging first. I did it in the wrong order and got lucky.

What I deliberately didn't build

The Claude Code ecosystem already has heavier answers to context loss: full memory layers with MCP tools, BM25 retrieval over past sessions, three-file dev-docs systems you maintain by hand. Good tools, different trade-offs.

compact-ops stays intentionally narrow: no MCP server, no database, no new workflow to learn, no daemon. It's shell scripts behind official hooks, and state is plain markdown you can cat. If it ever misbehaves, you uninstall the plugin and you're back to stock Claude Code -- there's nothing to migrate out of. I wanted the insurance to cost nothing when it isn't paying out.

The other non-feature: it never replaces the built-in summary. Tools that intercept or rewrite compaction itself break whenever Claude Code changes internals. Hooks are the supported surface, so that's the whole footprint.

Try it

git clone https://github.com/kenimo49/compact-ops.git
claude plugin marketplace add /path/to/compact-ops --scope user
claude plugin install compact-ops@compact-ops-local

Requirements: Claude Code v2.x, jq, and the claude CLI as the state-writing backend. Linux and macOS. After installing, just run /compact as usual -- each compact costs one extra LLM call (Sonnet by default, downgradable to Haiku or off).

The plugin's first beneficiary was the session that built it: saved by its own 67% warning, then stress-tested by its own hook failure, in the same night. Agent tooling is like that -- the reasons for a design only get written down where something actually broke.

What does your agent forget after /compact? If you have a story where a post-compaction session confidently undid your afternoon, I want to hear it.

7 Services, 0 Upload Solutions: Why MCP File Transfer Fails

Ken Imoto — Mon, 06 Jul 2026 13:05:10 +0000

I spent a day attaching a single receipt to a freee expense through an MCP client. It never worked. So I went wider: seven services my team actually uses in production, one upload test each.

Result: zero of seven support file upload in the standard MCP way. Four refuse it outright. Three "work" only because the MCP server is reading files off its own local disk — which falls over the moment you containerize the server, and which is not really MCP at all.

This is not a server bug list. The hole is in the protocol.

The protocol has 3 content types. File is not one of them.

MCP tool results return one of three content types:

TextContent (JSON strings, structured data)
ImageContent (base64-encoded PNG / JPEG only)
EmbeddedResource (URI reference to a host-owned resource)

There is no FileContent. There never was. An Anthropic engineer answered a public Discussion #1197 thread on this directly:

"I don't think you're overlooking anything, your use-case is currently finicky in the current state of the protocol."

Translated from politeness: this is a real gap, you found it, we know.

SEP-1306 proposed adding a binary elicitation mode in August 2025, where the server hands the client an upload URL and the client POSTs multipart/form-data to it. A later proposal, SEP-2356, adds Tool.inputFiles and requestedFiles so tools can declaratively say "I expect files here" and clients can render a native file picker.

Both are proposals. Neither is in the 2026-07-28 release candidate. The release candidate ships Apps, Tasks, and elicitation form/URL modes — all of which are "do file work outside the core protocol" workarounds. The core itself stays text and base64-image only.

What each service actually does when you try

freee (accounting) — hard no

freee's REST API takes multipart/form-data receipt attachments. MCP's JSON-RPC layer cannot send that. The MCP server sits between your client and freee, and the protocol it speaks upstream simply has no slot for the bytes. Expense workflows that need receipt attachment do not complete end-to-end. They stop at "describe the receipt in text."

Jira / Confluence — hard no, and worse in containers

The Atlassian official remote MCP server's community response says it plainly: "file uploads or image attachments via the MCP Remote Agent are not supported."

mcp-atlassian Issue #618 is darker. The attachment tool expects a local filesystem path on the MCP server itself:

{
  "attachment_results": {
    "failed": [{
      "filename": "grafana.png",
      "error": "File not found: /home/user/jira-mcp/grafana.png"
    }]
  }
}

Your client has the screenshot. The MCP server is in a Docker container that cannot see your filesystem. Issue tracked, no fix path because the fix is upstream in the protocol.

Notion — on the roadmap, no date

Notion's docs state: "Image and file uploads are not currently supported in Notion MCP, but this is on our roadmap." Roadmap items without dates do what they always do.

GitHub — no, and developers have noticed

github-mcp-server Issue #738 asks for image upload so Claude Code can attach UI screenshots to PRs. The thread is closed-pending-spec. Translation: the GitHub team is also waiting for SEP-1306.

Gmail, Google Drive, Slack — "works" in a way that doesn't count

Third-party Gmail and Drive MCP servers attach files by reading the path off the server's own disk. CData's Slack server has an UploadFile tool that does the same. None of these are transferring bytes through MCP. They are transferring a path string, then the server opens the file locally.

Two problems with that:

The client and server need to share a filesystem. Containerize the server and the model is now describing files it cannot reach.
It's a side channel, not the protocol. If the spec eventually does add FileContent or binary elicitation, every "works today" path will be the wrong shape.

Why the core avoids binary on purpose

Worth steelmanning: the spec authors did not forget. The avoidance is deliberate, for three reasons that are all defensible in isolation and add up to a real product gap.

JSON-RPC is text-shaped. MCP picked JSON-RPC because it's introspectable and trivially debuggable. Binary in JSON means base64, which means a 33% size tax on every byte and a flood of tokens into the model's context window. A 1 MB receipt becomes 1.33 MB of text and probably ~340k tokens of nothing the model can do anything semantic with.

File paths are a known injection surface. Every "let the model send a filename" feature in the last 30 years has had a command injection or path traversal bug filed against it within a year. The MCP team is being conservative on purpose.

The context window is the bottleneck. If file upload were a primitive, every tool would want it, and conversations would routinely embed 5–10 MB of base64. The economics break before the security does.

These are real. They also mean the protocol is making the file problem someone else's. Right now that someone is every MCP server author, and the solutions don't compose.

The workaround that actually ships

Until SEP-1306 or SEP-2356 lands, the only thing that genuinely works end-to-end is the reference-not-bytes pattern:

The MCP client (or the user) uploads the file to object storage — S3, GCS, a presigned-URL endpoint your server owns.
The MCP tool call passes the resulting URL or storage key as a TextContent string.
The MCP server fetches the bytes from storage, hands them to the upstream API (freee, GitHub, Jira), and returns the new attachment ID as text.

Bytes never touch JSON-RPC. The model never sees a 340k-token base64 blob. The server is a thin proxy, and it works identically whether it runs on your laptop or in a container.

This is also, not coincidentally, the shape SEP-1306 will eventually formalize. Building this way now means the migration when (if) the spec lands is roughly: swap your hand-rolled upload endpoint for the spec'd one, keep everything else. It's also where I should have started, instead of spending the first day trying to convince the protocol to carry bytes it was never going to carry.

Five-second checklist before you commit to MCP for a file workflow

Does your upstream API need multipart/form-data? If yes, MCP is not your transport — it's your control plane. Wire object storage underneath.
Is your MCP server containerized? If yes, do not trust any "works locally" file path tool. It will silently fail in prod.
Is the file >1 MB? Don't even consider base64-embedding it in ImageContent. The token bill alone is a no.
Do you need an audit trail of which bytes were uploaded? Log the storage key, not the tool call — JSON-RPC didn't carry the bytes anyway.

What to take away

MCP is a great control plane. It is not a file transport. The 2026-07-28 release candidate confirms this for another release cycle. If file work is on the critical path of your agent — expenses, ticket attachments, PR screenshots, Notion uploads — you need object storage and a thin server-side fetcher, today. Treat the protocol's eventual FileContent as a possible future cleanup, not a roadmap dependency.

The seven-service test reads as a vendor problem until you sit with the spec. Then it reads as a spec problem with seven downstream casualties. Worth knowing which one you're solving.

If you want the systematic version — every MCP server I tested, the exact failure modes, OWASP MCP Top 10, and the production reference-not-bytes pattern with code — I wrote a Japanese book on it: MCP Security in Practice (Impress).

References

Anthropic Rewrote frontend-design Skill: 3 AI Design Clichés Named (With Hex Codes)

Ken Imoto — Fri, 03 Jul 2026 13:00:01 +0000

Anthropic quietly rewrote their frontend-design Skill on June 18 in commit 423563cf. The new version contradicts the old one on its central thesis, and names three specific AI-generated design clichés in the public plugin documentation. With hex codes.

The SKILL.md diff is +39 / -26 lines (commit-wide it's +41/-28, the rest is a marketplace.json bump and the plugin version going to 1.1.0). On paper, a maintenance bump. In reality, a philosophy reversal.

I noticed this while reviewing my own image-generation skills last week against the upstream anthropics/claude-code version of the file. It slid in under the radar, which is a shame because it's one of the more interesting design-engineering shifts Anthropic has shipped recently. Here is what changed and why it matters if you ship UI that touches a model.

Old version: be bold everywhere

The old Skill's central instruction was extreme. From the file Anthropic had been shipping until June 18:

Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined, playful/toy-like, editorial/magazine, brutalist/raw, art deco/geometric, soft/pastel, industrial/utilitarian, etc.

And closed with:

Don't hold back, show what can truly be created when thinking outside the box and committing fully to a distinctive vision.

The structure of the old document was a list of axes to push: Typography, Color, Motion, Spatial Composition, Backgrounds. Each axis got its own "be distinctive here" paragraph. The implicit instruction was: maximize boldness across every axis simultaneously.

It reads like a pep talk. Imagine running it on every UI generation in your product. Now imagine the output.

New version: be bold in exactly one place

The new Skill opens with a completely different frame:

Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's. This client has already rejected proposals that felt templated, and is paying for a distinctive point of view: make deliberate, opinionated choices about palette, typography, and layout that are specific to this brief, and take one real aesthetic risk you can justify.

Note the word one. The new file uses it again later, more directly:

Spend your boldness in one place. Let the signature element be the one memorable thing, keep everything around it quiet and disciplined.

And then closes the restraint section with an aphorism widely attributed to Coco Chanel:

Consider Chanel's advice: before leaving the house, take a look in the mirror and remove one accessory.

The reversal is precise. Old: pick an extreme on every axis. New: pick one signature, keep the rest quiet, then remove one more thing before you ship.

If you read both versions back to back, the old one reads (to me) like it was written for the demo. The new one reads like it was written by someone who has sat with a year's worth of "be bold everywhere" outputs and noticed they all converged. I have no insider information on intent — this is just how the two documents land if you read them in sequence.

They named the three defaults. With hex codes.

This is the part I did not expect.

The new file contains this paragraph:

AI-generated design right now clusters around three looks:
(1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent;
(2) a near-black background with a single bright acid-green or vermilion accent;
(3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns.

Anthropic is publicly stating, in their own plugin docs, the hex value of the cream background their model defaults to. They followed it with this carefully worded line:

All three are legitimate for some briefs, but they are defaults rather than choices, and they appear regardless of subject.

That sentence is doing real work. It is not saying "these are bad designs." My read is closer to: "these are what we ship when no one is steering us." Either way, the team is auditing their own output and writing the audit into the public docs.

This is essentially the visual-design equivalent of the AI Slop word lists that text-side teams have been maintaining for a year. To make the parallel concrete, here is what the three defaults look like rendered as actual product hero sections. Same fictional product (an audit tool called Lumen, naturally), three default treatments.

Cliché 1: #F4F1EA cream, a Playfair-style italic serif, terracotta accent. Reads "editorial sophistication" at a glance, reads "every AI-generated landing page I've seen this quarter" two seconds later.

Cliché 2: near-black background, single bright accent, monospace details, "trusted by" strip. First impression: edgy modern tech. Closer look: indistinguishable from the last three YC Demo Day landing pages you visited.

Cliché 3: broadsheet style, hairline rules, zero border-radius, dense columns. Signals "serious and intellectual," then immediately falls back into the AI startup "About" page genre.

None of these are bad. They are all competent, defensible, ship-ready. They are also default behavior, which means they read as templated regardless of what the underlying product actually does.

The process got replaced with a loop

The other major structural change is in how the Skill instructs the model to work. The old version had a list of axes. The new version has a process.

Process: brainstorm, explore, plan, critique, build, critique again

The expanded instructions describe a five-step loop:

Read the brief. If it is vague, pin a subject, audience, and the single job the page must do.
Build a compact token system: 4-6 hex values, 2+ type roles, a layout described in prose + ASCII wireframe, and a signature element.
Critique the plan against the brief. Anywhere it reads like "the generic default you would produce for any similar page," rewrite it and say what changed.
Build it.
Take a screenshot and critique your own output.

This is, structurally, code review applied to design. Plan → diff against the spec → implement → self-review. The Skill is essentially asking the model to do its own design critique as a first-class step, with explicit instruction to flag any place where it would have produced the same thing for any other brief.

Whether the model can actually do this self-critique reliably is a separate question. But the intent — "audit your own defaults as part of the work" — is a notable shift from "execute the brief."

Copy got promoted to design material

The other new section is "More on writing in design." It did not exist in the old version. The opening line is:

Words appear in a design for one reason: to make it easier to understand, and therefore easier to use. They are design material, not decoration.

The rules are practical. Some I want to lift directly:

A button says exactly what happens. "Save changes," not "Submit."
The same verb threads through the whole flow. The button labeled "Publish" produces a toast that says "Published."
Errors don't apologize. They state what happened and how to fix it.
An empty screen is an invitation to act, not a mood.

If you have ever fought with a product team about whether UX copy is "the designer's job" or "the engineer's job," Anthropic just put it on the design Skill's responsibility list.

What restraint actually looks like

Here is the same Lumen hero, rebuilt with the new Skill's philosophy. Navy monochrome, one signature element: the word "One." set enormous, taking the entire vertical. Everything else quiet and disciplined.

The "after" treatment. Navy text on near-white, the single word "One." set giant and italic as the only visual moment, navigation and footer deliberately small. The image's job: prove that "spend boldness in one place" is a real layout choice, not a slogan.

The signature here is a typographic moment, but the recipe generalizes: pick one axis (color, type, layout, motion, decoration) to push hard, then withdraw on every other axis. The trap the old Skill set was telling the model to push on every axis at once, which paradoxically forces convergence on the safest combination of attacks. The new Skill explicitly diagnoses this in the line "Spend your boldness in one place."

What this means if you ship UI through a model

A few practical takeaways.

Check your generation prompts and Skills against the new version. If your prompt has language like "be bold," "be distinctive," "push the design," you are likely producing one of the three named clichés. The fix is to specify one axis to push and explicitly constrain the rest.

The hex-code audit is borrowable. "If the output background is near #F4F1EA and the accent is terracotta, flag it" is a check you can actually write. The same shape works for the other two clichés (near-black with an acid-green accent, broadsheet hairlines on white) — pick your own threshold values for those, Anthropic only spec'd the cream one to the hex. It's the visual equivalent of grep-ing for "delve" in LLM output.

Bring copy into the design pipeline. If your design tooling doesn't include button labels, empty-state text, and error messages as first-class artifacts, you're shipping a stale split of responsibilities. Anthropic just promoted copy to "design material" in their public docs.

Build a critique step into your generation loop. The "plan, build, critique, build again" pattern is portable. You can wrap any single-shot generation in a self-review pass that explicitly asks: "what would you have produced for any similar brief, and how does this differ?"

The summary

If I had to compress the new Skill into one sentence: AI-generated design fails by being bold everywhere; the fix is to be bold in exactly one place and remove one accessory before shipping.

Anthropic is auditing the defaults their own model produces and writing the audit into the public plugin docs. That's a useful thing to read, and an unusually honest move to make where everyone can see it.

The full diff is at commit 423563cf if you want to read it cold. It's worth ten minutes.

If you want the systematic version of building Skills like this — how SKILL.md is structured, how to compose Skills with subagents, when to use a Skill vs. a slash command — I wrote a Kindle book on it: Claude Code Mastery.

References

MCP Servers Ship Without OAuth. I Added It and 3 of 5 Clients Broke.

Ken Imoto — Thu, 02 Jul 2026 13:00:01 +0000

The honest version of my MCP server's auth story is short: there wasn't any. The server ran over HTTP, anyone who knew the URL could call its tools, and the tools touched a real account. I knew this was bad. I had even written about other people doing it. So I did the responsible thing and added OAuth 2.1 the way the MCP spec describes it.

Then I connected the same server from five clients to confirm nothing broke. Two of them authorized cleanly. Three of them broke, each in a completely different place, and not one of the failures was a bug in my server. The spec was satisfied. The clients had simply not caught up to it. That gap, the few inches between "the spec says X" and "the client does X," is exactly where your authorization lives, which means it is exactly the place you cannot afford a gap.

What "adding OAuth to an MCP server" actually means in 2026

Adding OAuth to an MCP server in 2026 means implementing the 2025-06-18 authorization spec, where your server is an OAuth 2.1 Resource Server, not its own authorization server. This is the part that surprised me, because it inverts the older mental model.

In the previous revision (2025-03-26) the MCP server was treated more or less as its own authorization server. The 2025-06-18 changelog reclassified it: "Classify MCP servers as OAuth Resource Servers, adding protected resource metadata to discover the corresponding Authorization server." So my server's job got smaller and pickier at the same time. It no longer issues tokens. It validates them, advertises where the real authorization server lives, and rejects anything addressed to someone else.

Concretely, a compliant server has to do four things:

Return 401 Unauthorized with a WWW-Authenticate header pointing at its resource metadata URL.
Serve Protected Resource Metadata at /.well-known/oauth-protected-resource (RFC 9728), naming the authorization server.
Let the client discover the authorization server's own metadata (RFC 8414) and register through Dynamic Client Registration (RFC 7591), which the spec marks as SHOULD, not MUST.
Validate the token's audience so a token minted for some other service can't be replayed against mine (RFC 8707 resource indicators).

I implemented all four. I ran the MCP Inspector against it and watched the full dance: 401, metadata fetch, browser redirect, token, authorized call. Green across the board. That was the moment I assumed I was done, which is the exact moment in every war story where the narrator is about to learn something.

The test: five clients, one server, same handshake

My test was deliberately boring. One server, five clients, the same OAuth handshake each time. The clients were the ones I actually use or expect my users to use: Claude Desktop, VS Code with the Copilot MCP integration, Cursor, Claude Code, and a remote bridge (mcp-remote) for anything that only speaks stdio.

I went in expecting an even spread of minor papercuts. What I got was a clean split. Two clients walked through the front door. Three got stuck in three different doorways, and the interesting part is that the three failures map almost perfectly onto the three things the spec leaves optional or underspecified. The clients didn't fail randomly. They failed exactly where the spec gives them room to disagree.

Here is the scoreboard before the autopsies.

Client            Result    Failed at
----------------------------------------------------------
Claude Desktop    PASS      (clean)
MCP Inspector     PASS      (clean)
VS Code (Copilot) BROKE     loopback redirect URI port
Cursor            BROKE     dynamic client registration
Claude Code       BROKE     manual client_id rejected

(MCP Inspector is a tool, not a shipping client, but it earned its row by being the only thing that worked on the first try, which tells you something about who the spec was written against.)

Break #1: VS Code and the loopback port that wasn't there

VS Code broke because its OAuth callback wanted a specific loopback port that was already taken, and the authorization server refused the redirect. The error on screen was "Invalid Redirect URI," which is an unhelpful thing to read when your redirect URI looks correct.

VS Code added native MCP authentication in version 1.101: it parses WWW-Authenticate, follows the metadata, and tries DCR first. All correct. The trouble is the redirect target. OAuth for a desktop client uses a loopback redirect (http://127.0.0.1:<port>/...), and RFC 8252 §7.3 is explicit that the authorization server MUST allow any port for loopback, because the client picks an ephemeral one at runtime. Plenty of authorization servers ignore that and do strict, exact redirect-URI matching including the port. When the port VS Code grabbed didn't match what was registered, the handshake died.

This is not a hypothetical I reverse-engineered from my own logs. It is filed: microsoft/vscode #278512, "MCP: OAuth Server Redirect URI Mismatch Bug," describes the default loopback port being unavailable and producing exactly "Invalid Redirect URI." The same shape shows up in github/copilot-cli #1951 and in claude-code #51319, where a hardcoded callback port breaks the moment you run two sessions at once. The fix lives on the authorization server: relax redirect matching for loopback the way the RFC tells you to. But you only learn you needed that fix by watching a client fail, because the spec-compliant server and the spec-compliant client can still disagree about whether "port" is part of the URI.

Break #2: Cursor and the authorization server that wouldn't register a stranger

Cursor broke because it insisted on Dynamic Client Registration, and my authorization server, like most authorization servers, doesn't offer it. DCR is the step where a brand-new client introduces itself to the authorization server at runtime and gets credentials on the spot, no human pre-registration. The MCP spec lists DCR as SHOULD. Clients read SHOULD as "rely on it." Authorization servers read SHOULD as "skip it."

That mismatch is the single most common way OAuth-enabled MCP servers break, and it is not subtle in the issue trackers. github/github-mcp-server #1404: "Dynamic Client Registration not supported." claude-code #38102: the client "attempts DCR even when a clientId is supplied." openai/codex #15818: OAuth login "fails when authorization server does not support dynamic client registration." microsoft/vscode #279955, filed flatly as "Dynamic Client Registration (DCR) issues in MCP Servers." Every major client has a version of this.

The root cause is an asymmetry the spec bets on: it assumes authorization servers will support DCR, and most of them do not. Okta, Entra ID, and the rest of the enterprise identity world mostly want you to pre-register a client by hand. So a client that only knows how to DCR is a client that cannot talk to the authorization server every enterprise already runs. I had wired my server to a real identity provider, not a toy one, and the toy was the only thing that would have worked unmodified.

Break #3: Claude Code and the client_id it refused to use

Claude Code broke in the mirror image of break #2: I tried to hand it a pre-registered client_id to dodge the DCR problem, and it tried to do DCR anyway. The whole reason to support a manual client_id is to work with authorization servers that don't do DCR. A client that ignores the client_id you gave it and reaches for DCR regardless has, in effect, no escape hatch.

This one is also documented rather than imagined. claude-code #38102 is titled, almost word for word, "MCP OAuth: 'does not support dynamic client registration' despite clientId configured." On the VS Code side the gap is tracked as a feature that doesn't exist yet: #252892, "capability to register a clientId for MCP OAuth," and #257415, "No option to disable Dynamic Client Registration and use custom (static) client information." Anthropic's own connector docs say the quiet part directly for Claude.ai: it "requires Dynamic Client Registration support and does not yet support a way for users to specify a client ID or secret."

So the two failures are a pincer. Break #2 is a client that demands DCR from an authorization server that can't do it. Break #3 is a client that won't accept the manual credential you'd use to route around DCR. Same root, opposite ends, and between them they cover most of the identity providers a real company actually uses.

How I got from three breaks to zero

I got from three breaks to zero by fixing the authorization server, not the clients, plus one bridge for the client I couldn't fix. The clients aren't mine to patch, so the move was to make the server tolerant of the ways clients disagree.

For the loopback port mismatch: relax redirect-URI matching on the authorization server so any loopback port validates, per RFC 8252. This is a server-side allowance, and it's the one change that quietly fixes a whole family of "Invalid Redirect URI" reports at once.
For the DCR-required client against a no-DCR authorization server: put an OAuth proxy in front that does speak DCR, registers the client dynamically, and maps it onto a pre-registered upstream client. This is precisely why OAuth proxies exist in the MCP world, and why the community keeps building them.
For the client that ignored my manual client_id: route it through mcp-remote, the npx bridge that runs the OAuth flow on the client's behalf and, in its static-config form, lets you inject a pre-registered client_id the picky client never had to know about.

None of these are elegant. They are shims around the seam between a spec and its implementations. But they share a useful property: every fix lives on infrastructure I control, because the one thing you can't ship is "please upgrade your client."

What I'd tell you before you add OAuth to an MCP server

Before you add OAuth to an MCP server, assume the spec is the easy part and client compatibility is the actual project. I budgeted my time backwards. The server-side OAuth 2.1 implementation took an afternoon. Reconciling it with how five clients actually behave took the rest of the week, and that ratio is the real lesson.

Three things I'd bolt to the wall:

Test against real clients, not just the Inspector. The Inspector is a reference implementation and it passes by construction. It is the most optimistic possible reader of your server. Your users are running Cursor and VS Code and Claude Code, and those are the readers who disagree. A green Inspector run means your server is correct, not that anyone can connect to it.

Treat DCR as a coin flip, and own both sides. Half the clients assume Dynamic Client Registration works; half the authorization servers don't offer it. You cannot pick a side and be safe, because your users bring their own clients and your company brings its own identity provider. The durable answer is a proxy that speaks DCR outward and pre-registration inward, so the disagreement resolves on your infrastructure instead of in an error dialog.

Know which spec revision your clients are reading. The auth spec is moving fast. The 2025-11-25 revision already deprecates DCR in favor of Client ID Metadata Documents and makes PKCE mandatory, which means a year from now the breakage will have a new shape and the same cause: clients catching up to the spec at different speeds. The thing that broke three of my five clients wasn't a flaw in OAuth or in MCP. It was the lag, and lag is permanent.

If you want the full map of where MCP's attack surface actually lives, including why so many servers ship with no auth at all and what the OWASP MCP Top 10 says to do about it, I wrote MCP Security in Practice. This post is the chapter that only exists because I tried to fix one of those gaps and found three more.

The takeaway

Adding OAuth to an MCP server is not a single task; it is a server task you finish in an afternoon followed by a compatibility task you finish never, because the clients keep moving. My server was spec-correct the whole time. Three of five clients still couldn't connect, because each one disagreed with the spec in a different place: a loopback port, a missing DCR, a rejected client_id. The fixes all lived on my side, which is the good news and the bad news at once. The good news is you can fix it. The bad news is "it works in the Inspector" was never the finish line. The finish line is five clients, and the spec only gets you to the door.

RTX 4070 + Qwen 35B: 2.8x Speedup From One llama.cpp Flag (--cpu-moe)

Ken Imoto — Wed, 01 Jul 2026 13:00:01 +0000

The Ollama defaults gave me 12.2 tok/s on Qwen3.5-35B-A3B against an RTX 4070 (12 GB). I switched to llama.cpp with two flags and got 34.6 tok/s. 2.8x.

The two flags were -ngl 99 (offload all layers to GPU) and --cpu-moe (except the MoE experts, which go on the CPU). One of them is the obvious one. The other is the entire post.

llama-server -m qwen35.gguf -ngl 99 --cpu-moe -c 4096

I want to walk through why that specific combination works on a 12 GB card that "should not" fit a 35B model, the full offload sweep so you can pick the right setting for your VRAM tier, and the three things that will cause the same command to under-perform on your machine.

The setup, briefly

GPU: RTX 4070, 12 GB
RAM: 31 GB
OS: WSL2 Ubuntu 24.04
CUDA: 12.9
Model: Qwen3.5-35B-A3B, Q4_K_M quantization (20.49 GiB, 34.66B params, 128 experts, 8 active per token)
llama.cpp: built with cmake -B build -DGGML_CUDA=ON

If you are on the newer Qwen3.6-35B-A3B (released early 2026) or the smaller Qwen3-Coder-30B-A3B, the same flag math applies. Both are 128-expert MoEs with 8 active per forward pass, so the offload ratio is identical and the speedup pattern transfers.

Why "no chunks in VRAM" wins on a 12 GB card

The reflexive instinct on a 12 GB card is to pack as much of the model as possible into VRAM and let the rest spill to RAM. That instinct is wrong for MoE.

A dense 35B model wants every layer warm because every weight participates in every token. A MoE model is the opposite: of 128 experts in each layer, only 8 fire per token. The other 120 are dead weight that round trips for nothing.

-ngl 99 says "put every layer on the GPU." On a 12 GB card with a 20 GiB model, this should be impossible.

--cpu-moe (added to llama.cpp in mid-2025 as the all-CPU shortcut for the more granular --n-cpu-moe N) is the escape hatch: put every layer on the GPU except the MoE experts, which stay on the CPU. Now what is on the GPU is the bandwidth-hungry part (attention, KV cache, layer norms, the router) and what is on the CPU is the sparse part (the experts themselves, which barely fire).

The result on this machine:

VRAM used: 11.7 GB (95% of 12 GB)
Generation: 34.6 tok/s (vs Ollama default 12.2)

Ollama's autoloader is not dumb — it figures out that ~58% of the model has to go on CPU and ~42% on GPU. But it splits the model the dense way: by layer. So you end up with half the attention paths on CPU (where bandwidth chokes) and half the experts on GPU (where they sit idle most of the time). It is the worst of both worlds, and it costs you 2.8x.

The full offload sweep

The flag people will reach for next is --n-cpu-moe N, which lets you offload only N layers of experts to CPU and put the rest on GPU. The instinct is "well, GPU is faster, so put as many experts back on GPU as fit." This is also wrong, and the sweep shows by how much.

-ngl 99 is fixed. Only N changes:

`--n-cpu-moe`	Expert layers on GPU	tok/s (tg128)	vs Ollama default
48 (all experts on CPU)	0	34.60	2.8x
44	4	27.19	2.2x
40	8	16.88	1.4x
36	12	15.29	1.3x
32	16	14.06	1.2x
28	20	12.85	1.1x
24	24	11.71	0.96x

Monotonic decay. The moment you start pulling experts back onto the GPU, you steal VRAM from the parts that actually need bandwidth, and the whole pipeline slows down. By the time half the experts (24 layers) are back on GPU, you are slower than Ollama's automatic split.

The reading is direct: the optimum for a 12 GB card on a 128-expert MoE is all experts on CPU. Not "as many as fit." All of them.

The bench command to reproduce, if you want to:

./build/bin/llama-bench -m qwen35.gguf -ngl 99 -ncmoe 48 -n 128 -r 3

-ncmoe 48 is the bench-tool equivalent of --n-cpu-moe 48, which in turn is the explicit form of --cpu-moe for a 48-layer model. Same setting, three names. The flag landed in llama.cpp during a period of active iteration on MoE offload semantics.

What VRAM tier maps to what

You can almost read your sweet spot off the table above. Roughly:

8–10 GB cards (RTX 4060 / 3070): full --cpu-moe. You will not have headroom to put any experts on GPU.
12 GB cards (RTX 4070 / 3060 12 GB): full --cpu-moe. The sweep above is yours. 34.6 tok/s is the realistic ceiling for Q4 35B-A3B.
16 GB cards (RTX 4060 Ti 16 GB / 4070 Ti SUPER): you can start putting a few expert layers back on GPU (N=44 → 4 layers on GPU) and gain a little, but only a little — the next regime gets dominated by PCIe-bus expert transfers that eat the win.
24 GB cards (RTX 4090 / 3090): you can fit the model fully and skip this entire post. Lucky you.

The crossover where "all on GPU" beats "experts on CPU" is somewhere around 24 GB for Q4 35B-A3B. Below that, the math says spread.

Three things that will tank your number

The 34.6 tok/s above is not what you get by pasting the command. It is what you get by pasting the command after clearing three traps.

Trap 1: VRAM is not actually free. WSL2 happily shares VRAM with Windows-side processes. If Edge has 200 MB of "hardware acceleration" stuck in VRAM, your attention layer fights for it and loses. Check with nvidia-smi before benching. The number you want next to "Used" is under 500 MB before you start.

Trap 2: Qwen thinking mode. Qwen3.5 has a "thinking" mode that emits reasoning tokens before the answer. If you benchmark with a generic prompt, it will think for 400 tokens about "what is 2+2" and your tok/s number is meaningless. Either disable thinking via the system prompt or measure with prompts that bypass it.

Trap 3: Quantization and build flags. The 34.6 figure is Q4_K_M with a CUDA-enabled build. Q5_K_M will fall to roughly 28–30 tok/s on the same card because the experts get heavier per token. A CPU-only build of llama.cpp will obviously sit at single digits. If your number is off by 40%, check nvidia-smi during inference — llama-server should show 95%+ GPU utilization on the prompt and 30–60% on generation. If it shows 5%, you are running CPU-only without realizing it.

The line worth memorizing

For dense models, "put as much as possible on GPU" is correct. For sparse MoE on consumer GPUs, "put everything on GPU except the experts" is correct, and the gap between those two heuristics is 2.8x on this card.

The one-line version: the bottleneck on a 12 GB card running a 35B MoE is not parameters. It is bandwidth. The right partition puts the bandwidth-hungry part on the bandwidth-rich device, even when that means leaving 60% of the parameter count on the CPU.

If you take one thing away from this post: run llama-bench with -ngl 99 -ncmoe 48 -n 128 -r 3 on your card and write the number down. If it is more than 2x your Ollama default, the rest of your local-LLM tuning is variance. If it is less, your VRAM is not actually free.

If you want the full data-driven engineering pattern behind this kind of measurement-first tuning — same logic applied to broader system harnesses, build pipelines, and agent loops — Harness Engineering Guide is the long form.

Barge-In Is the Voice-Agent Feature Nobody Benchmarks. I Added It and Lost 120ms.

Ken Imoto — Tue, 30 Jun 2026 13:00:01 +0000

I have read more voice-agent benchmarks than I would like to admit. They all measure the same thing: how many milliseconds from "user stops talking" to "agent starts talking." Stack comparisons, P95 charts, the whole genre. Every one of them treats the conversation as a relay race where only one runner is moving at a time.

Then I shipped barge-in: the ability for a user to talk over the agent and have it shut up gracefully. And I discovered the thing none of those benchmarks measure. Letting the user interrupt is not free. On my own pipeline, turning it on added 120ms to the exact latency number every chart obsesses over. Nobody benchmarks the cost of barge-in, because barge-in is the feature that makes your headline number worse, and no one wants to publish that.

This is the field report.

What barge-in actually is

Barge-in is the agent yielding the floor when the user starts speaking mid-response. It is the difference between a conversation and a press conference. Without it, the agent finishes its sentence no matter what you do, and you end up talking over a machine that cannot hear you over itself.

Here is the part that surprised me. Every benchmark I had read measures a half-duplex world: the user talks, then the agent talks, cleanly alternating. Barge-in breaks that model entirely. To support it, the agent has to keep listening while it is speaking. That is full-duplex, and full-duplex is where the hidden cost lives.

The numbers that get published are turn-taking latency: human conversation hands off in roughly 200-300ms, while most agents land somewhere between 800ms and 1500ms (Gradium, Softcery). Those are the numbers in the charts. The number nobody charts is how long it takes the agent to stop once you cut in.

The thing I shipped, and the thing it broke

My setup is a fairly ordinary cascade: WebRTC transport, streaming STT, an LLM, streaming TTS, with an orchestrator wiring the frames together. End-to-end I was sitting in a respectable place. Then the complaints came in, and they were the same complaints everyone gets: the agent steamrolls you, it answers before you finish, it keeps talking when you try to correct it.

So I added barge-in. The mechanism is not exotic. While TTS is playing, you keep a VAD running on the inbound mic stream. The instant it fires, you duck the agent audio and decide whether to yield the turn. The common production move is to drop TTS gain by about 24dB the moment VAD fires, without killing the stream, so you can recover if it was a false alarm (Future AGI).

It worked. The agent stopped steamrolling. And my latency went up, because of the part I had not thought hard enough about: false barge-ins.

Why naive barge-in is a disaster

The first version fired on everything. A cough. A door. The user saying "mm-hm" to agree, which is not an interruption at all, it is a backchannel. Worst of all, it fired on the agent's own voice leaking back through the mic. The agent heard itself, decided someone was interrupting, and went silent. A machine startled by its own echo, like a dog barking at the mirror and then losing the staring contest.

The fix for the echo problem is acoustic echo cancellation: feed the speaker output back as a reference signal, subtract it from the mic input, and you are left with just the human. That is table stakes for full-duplex and I will not relitigate it here.

The fix for the cough-and-backchannel problem is where the latency went. You cannot trust a single VAD frame. Energy-based VAD does not know the difference between "I disagree, stop" and someone clearing their throat in a coffee shop. Background noise pushing energy above threshold is exactly the failure mode the field keeps naming (Future AGI). So you add a guard. You require the interrupting speech to persist for a minimum duration before you commit to yielding.

That guard is the 120ms. And it buys something real. A minimum-duration guard can cut the false-barge-in rate by 60-80%, but it adds roughly 200ms to the barge-in path (Future AGI). I tuned mine tighter than that and landed at +120ms before my false-positive rate dropped under the 5% I was aiming for. The published target for barge-in is brutal in both directions: 95%+ accuracy, under 5% false positives, under 5% missed real interruptions (Future AGI). You do not get there for free, and the currency you pay in is the same milliseconds your benchmark is bragging about.

The two timers nobody puts on the same chart

Here is what I think the benchmarks get structurally wrong. There is not one latency in a voice agent. There are two, and they pull in opposite directions.

Timer	What it measures	Direction barge-in pushes it
Turn-taking latency	User stops -> agent starts	This is what every chart reports
Barge-in latency	User cuts in -> agent stops	This is the one nobody reports

Turn-taking latency is the relay-race number. Barge-in latency is the interrupt-handling number, and the field is starting to put real targets on it: interruption response under 200ms, measured from user-speech onset to TTS suppression (Future AGI). The trap is that these two timers fight. Make the agent quicker to yield and you generate more false stops. Add a guard to kill the false stops and you slow the yield. You are not optimizing a number. You are choosing a point on a tradeoff curve, and the benchmark that reports only the first timer is hiding the second axis entirely.

The research framing I found most honest measures the minimum latency required to reach 90% barge-in accuracy, rather than reporting latency and accuracy as if they were independent (Future AGI). That is the joint metric. That is what a barge-in benchmark should look like, and almost nobody publishes it.

Where the +120ms actually fits in the budget

To be clear about scale: in a cascade, the latency that gets all the attention is the STT-to-LLM-to-TTS chain, which even at its fastest is a few hundred milliseconds of stacked work. The barge-in path is a separate budget. It runs in parallel, on the listening side, the whole time the agent is talking. The response chain and the listening path never touch.

So the +120ms does not lengthen your response. It lengthens the interruption. When a user cuts in, that is the delay before the agent goes quiet. And that delay has a much lower tolerance than response latency does. People forgive an agent that takes 600ms to answer. They do not forgive an agent that keeps talking for 600ms after they have clearly told it to stop, because at that point it is not slow, it is rude. The barge-in timer is the one your users feel as a personality flaw.

What 2026 turn-taking models change

The honest version of this story is that the brute-force guard is the old way, and the field has moved. The fix for "VAD is too dumb to tell a cough from a correction" is to stop using a bare energy threshold and use a model that understands turns.

This is the shift everyone is making right now. The 2026 production stack is migrating from energy-threshold VAD toward dedicated turn-taking models that classify backchannel versus barge-in versus continued silence as a learned signal (Future AGI). The named players:

Deepgram Flux does model-native end-of-turn detection using acoustic, semantic, and conversational context instead of silence thresholds, landing around 250ms end-of-turn and removing the need for a separate VAD-plus-endpointing stack (Deepgram).
Krisp Turn Prediction v3 pushes end-of-turn latency below 200ms, and in May 2026 benchmarks its accuracy curve sat below LiveKit's built-in and Deepgram Flux's across the operating range (Krisp).
LiveKit Agents ships adaptive interruption handling at 86% precision and 100% recall, with the barge-in and backchannel-suppression logic living in the orchestrator, not the ASR model (Inworld).

That last point reframed the whole problem for me. Barge-in quality lives in your orchestrator, not your speech-to-text model. The model tells you what it heard; the orchestrator decides what to do about it, and that decision is the entire game (AssemblyAI). I had been tuning the wrong layer for a week.

A semantic turn detector earns back most of my 120ms because it does not need a long duration guard. It can tell that "actually—" is an interruption and "yeah, mm-hm" is not, from the prosody and the words, not from how long the sound lasted. The guard was a crutch for a dumb VAD. A model that understands the turn lets you commit to the decision sooner with the same accuracy, which is the only way to move down the tradeoff curve instead of along it. Combining audio and text this way is what closes the gap to roughly 300ms without cutting users off mid-thought (Future AGI).

What I would tell myself before shipping it

Three things rearranged in my head, and they are the things I wish a benchmark had told me.

Measure the second timer. If your dashboard only has turn-taking latency, you are flying with one instrument. Add barge-in latency, measured from user-speech onset to TTS suppression, and watch them as a pair. The moment you start optimizing one in isolation, you are quietly wrecking the other.

The guard is a tax, not a feature. A minimum-duration guard is the cheapest way to stop false barge-ins and the most expensive way to feel responsive. It is fine as a first pass. It is a bad place to live. If you are still paying a 120-200ms guard tax six months in, you have not solved barge-in, you have postponed it.

Barge-in is an orchestrator problem. I spent days assuming a better STT model would fix my interruptions. It would not have. The yield-or-hold decision lives above the model, and that is where the engineering actually is. Pick your transport and orchestrator for how they handle interruption events, because that is the layer your users will judge.

The number nobody puts on the chart is the number your users feel first. An agent that answers fast but will not stop talking is not a fast agent. It is a fast bulldozer. I would rather lose 120ms and have it know when to shut up.

I pulled the latency-budget framing and the cascade anatomy behind this from my book The 300ms Voice-AI UX Problem, which is where I worked out why turn-taking is the part of the budget that does not behave like the rest of it. This post is what happened when I stopped reading about turn gaps and started measuring the one in the other direction.

When the Free Executor Cost More: 40 Trials on Opus + Local Qwen Ended Up the Most Expensive Cloud Arm

Ken Imoto — Sat, 27 Jun 2026 23:42:17 +0000

"Use a strong model to orchestrate, a cheap model to execute." This is now the default cost-aware recipe for agentic coding.

I believed it and ran the experiment. Opus 4.7 as the orchestrator, locally-hosted Qwen 3.5-9B (zero token cost) as the executor. This should beat running Opus alone on cost. Has to.

It did the opposite.

The supposedly "free" configuration (Opus + Qwen) came out as the most expensive cloud arm on all three of the code-repair tasks I ran. Higher than Opus solo. Higher than Opus + Haiku. And of course much higher than Haiku solo. As someone who actually built a GPU PC believing "local means cheap," I find this somewhat inconvenient.

I wrote up the 40 trials worth of numbers and the mechanism as a paper, archived on Zenodo: DOI 10.5281/zenodo.20978074 / GitHub repo.

This post walks through what happened across the 40 trials and why "free" turned out to be the most expensive option — all from real measurements.

TL;DR

40 trials × 4 configurations × 3 tasks, judged by a deterministic harness (mypy + ruff + pytest exit codes only). No LLM-as-judge anywhere in the loop.

Opus orchestrates + Qwen executes is the most expensive cloud arm on every task. More expensive than Opus solo.
The cause is not the executor's tokens — it's the orchestrator's prompt-cache re-reads. Opus keeps reading Qwen's returned summaries on every turn, and its own input volume grows to 1.4–5.3× that of Opus running alone.
Haiku solo is 5.5× cheaper than Opus solo on the largest task — but fails 25% of the time within the per-arm iteration cap. Within cloud-only options, Opus + Haiku is the most balanced.

If the intuition "the executor's tokens are free, therefore this is cheap" feels obvious, this post is about why that intuition breaks.

What I Measured

The four arms

arm	orchestrator	executor	role split
A	Opus 4.7	(solo)	one model does everything
B	Opus 4.7	Qwen 3.5-9B (local / Ollama)	Opus plans + verifies, Qwen edits
C	Opus 4.7	Haiku 4.5 (Anthropic SDK sub-loop)	Opus plans + verifies, Haiku edits
D	Haiku 4.5	(solo)	one cheap model does everything

All four arms use the Anthropic SDK with the same tool surface: str_replace_editor (view/create/str_replace/insert) and a bash tool with a 120-second timeout. The orchestrator arms (B, C) get one extra tool: delegate_to_executor.

Anthropic prompt caching is enabled identically on every call — system, tool definitions, and the most recent user message are marked with cache_control: ephemeral. No temperature or seed is set, so trial-to-trial variance reflects sampling noise.

The three tasks

All three operate on the typer repository at commit b210c0e (v0.26.8, MIT license). Each trial starts with git checkout -- . && git clean -fd to restore the base state.

T1 — Breakage recovery: 25 errors injected via AST (10 mypy + 10 ruff + 5 pytest collection failures). The agent has to return the harness to fully green.
T2 — Refactor: Move get_params_from_function from typer/utils.py to a new module typer/_param_extractor.py. Update every import site. All tests still passing.
T3 — Feature-add: Implement get_version_banner(prefix, uppercase) -> str, re-export from typer/__init__.py, and pass a SHA-256-fingerprinted test file.

The judge

mypy + ruff check + pytest — exit code 0 = success, anything else = failure. Per-task verifiers (verify-T2.sh / verify-T3.sh) add structural checks (function actually moved to the new module, fingerprinted test unmodified, etc.).

No LLM is ever asked "is this OK?". The judgment is deterministic and reproducible.

Results (success-only medians, n=3 per cell)

arm	task	n_succ/total	wall (s)	iters	cost ($)	success rate
A Opus solo	T1	3/3	253	36	1.74	1.00
A Opus solo	T2	3/4	233	26	1.11	0.75
A Opus solo	T3	3/3	69	6	0.17	1.00
B Opus+Qwen	T1	3/4	484	38	2.27	0.75
B Opus+Qwen	T2	3/3	443	27	1.38	1.00
B Opus+Qwen	T3	3/3	348	12	0.42	1.00
C Opus+Haiku	T1	3/3	400	28	1.67	1.00
C Opus+Haiku	T2	3/3	275	20	0.92	1.00
C Opus+Haiku	T3	3/3	145	11	0.38	1.00
D Haiku solo	T1	3/4	758	89	0.30	0.75
D Haiku solo	T2	3/4	507	70	0.23	0.75
D Haiku solo	T3	3/3	208	29	0.08	1.00

Bold = per-column winner. Total Anthropic spend across 40 trials: $35.98 — cheap for a paper.

The row worth staring at is arm B. On all three tasks, its cost ($) is the cloud-arm worst ($2.27 / $1.38 / $0.42). Qwen's tokens cost zero. Opus + Qwen is more expensive than Opus alone anyway.

Why "Free" Cost the Most

Compare Opus-side token consumption (input + cache_read_input) across arms:

arm role	T1 (Opus-side in + cache_r)	T2	T3
A (Opus solo)	534,586	226,474	13,320
B (Opus + Qwen)	733,142	313,914	62,864
C (Opus + Haiku)	421,622	159,640	44,016

B-over-A ratio (Opus-side only): 1.38× on T1, 1.39× on T2, 5.26× on T3.

Qwen's tokens are free. But Opus itself is reading 1.4–5.3× more tokens than it would running alone.

The mechanism. When Opus calls delegate_to_executor, Qwen returns a stdout summary (capped at 4000 chars in my implementation). That summary lands in Opus's context. Anthropic prompt caching marks the most recent message for cache_write, and the next turn reads it back via cache_read. Across 30–80 turns, Opus ends up reading the "what Qwen did" summary over and over and over.

Each re-read is billed at the cache_read rate ($1.50/M token = 10% of Opus input). The executor is free; the orchestrator is not. Which sounds obvious in hindsight, except the word "free" in a sentence tends to short-circuit human reasoning. Mine, anyway.

Stated correctly: the orchestrator's cost is proportional to how many times it re-reads the executor's returned summaries, not to the executor's raw token count. This reads more like a middle-management observation than an LLM finding, but the data says what it says.

Why T3 Blew Up to 5.3×

The most extreme case is T3, the smallest task — about 6 iterations.

Same mechanism, different ratio. The base context (system + tools + initial prompt) is cache_write-ed once on the first turn and cache_read cheaply thereafter. On long tasks (T1, T2), that base is a small fraction of the cumulative input. On a short task, it's a big fraction. So "base re-read every turn + executor summary re-read every turn" overhead dominates everything else, and T3's B/A ratio spikes to 5.3×.

Conversely, arm C (Opus + Haiku) has a smaller cache_read footprint than arm A on T1 and T2 (0.79× and 0.70× of A). Haiku does substantive work that Opus would have otherwise had to do itself, and the substance translates into useful summaries instead of dead weight. Which is the opposite end of the Qwen-summary-bloat story.

When Orchestration Would Win (cases I deliberately excluded)

The "strong orchestrator + cheap executor" recipe falters in iterative tool-loops because, over dozens of turns, the orchestrator's cache_read becomes the dominant cost line. One-shot routing has no such problem.

The experiment was, in that sense, designed against arm B:

Executor returns are free-form (Qwen stdout summary up to 4000 chars). If you constrain returns to "one structured diff and nothing else," the orchestrator's accumulated context shrinks.
Tasks are sequential (T1/T2/T3 cannot be parallelized within a single trial). Tasks where the orchestrator can dispatch "go edit these three places at once" might pay for orchestration overhead.

Re-running arm B with tightly-bounded executor returns is the next experiment on my list. I expect T3 to invert. T1 is harder to call.

Practical Takeaways

Sitting with these numbers, here is how my own agentic coding setup changed:

For tasks that finish in a handful of iterations, Opus solo is the cheapest cloud option. T3: $0.17, 69 seconds, 6 iterations — cloud-best. "Opus is expensive" is a one-shot framing. Across an iterative loop, Opus's per-iteration efficiency pays.
For tasks that need dozens of iterations, the model with the lowest per-iteration cost wins on dollars. T1: Haiku solo at $0.30 is 5.5× cheaper than the cheapest cloud arm. But it fails 25% of the time, so retry-adjusted expected cost narrows the gap to 4.2×.
For cloud-only setups, Opus + Haiku is the most balanced. Ties Opus solo on T1, wins T2 on cost, narrowly loses to Opus solo on T3. The safe pick if you don't want Haiku-solo's failure rate.
If you're going to use a local Qwen "for free," constrain the executor return size structurally. Free-form stdout returns just shift the cost to the orchestrator's cache_read line.

"Strong + cheap" composition has a narrower design surface than it seems. Unless you also specify what and how much the executor is allowed to return, you regenerate the "orchestrator-becomes-expensive" pattern. I regenerated it three times suspecting measurement error before finally accepting it.

Limitations

The honesty section:

n=3 per cell. Mann-Whitney U p-values use a normal approximation where 0.050 is the small-sample floor — it means "as different as this sample size can show." Trust the Cliff's delta effect sizes; don't over-read p-value differences.
All three tasks are on the typer repo. Generalization needs other codebases. The harness, breakage injector, runner, and analysis are all MIT-licensed in the repo, so reproducing this on your own codebase is cheap.
The orchestrator system prompt is asymmetric. It instructs "do not edit directly, delegate instead." This mirrors a real deployment shape but is a real confounder in the results.

Reproducing

Tested on Ubuntu 22.04 with Python 3.10+, uv 0.4+, and anthropic Python SDK 0.83+. Arm B uses Ollama 0.4+ running qwen3.5:9b. If you skip arm B, Ollama is not needed.

git clone https://github.com/kenimo49/free-executor-paradox
cd free-executor-paradox
# run arm A on T3, one trial
python scripts/runners/runner.py --arm A --task T3 --trial 1

The repo README and paper PDF have the full reproducibility setup — harness, breakage injection, runner, and analysis scripts are all included.

Closing

The cost debate around agentic coding tends to fixate on what the executor costs per token. The dominant term is actually what the orchestrator re-reads, and how often. Qwen here is just one instance of the pattern — every "free local executor" that comes next will hit the same issue. Free executor tokens don't make orchestrator cache_read free.

I wrote it up as a paper because numbers are harder to argue with than vibes. The most satisfying outcome would be someone replying "got the same thing on my codebase" or "actually got the opposite, here's why."

Paper (Zenodo): When Free Executors Cost More — DOI 10.5281/zenodo.20978074
Code + data + harness: https://github.com/kenimo49/free-executor-paradox
GitHub Release: v1.0.0

I Let Claude Code Run a Month of My Business Books. It Reconciled 200 Transactions and Miscategorized 11.

Ken Imoto — Thu, 25 Jun 2026 13:00:00 +0000

I run a few small businesses, which means once a month I sit down with a bank export and an accounting platform and turn a pile of transactions into something a tax office will accept. It is the part of self-employment nobody warns you about. The coding is fun. The books are not.

So this month I handed the books to Claude Code and watched.

The result, up front: it reconciled 200 transactions and miscategorized 11. That is a 94.5% hit rate, which sounds great until you remember that the 11 wrong ones were the difference between a clean filing and a letter from the tax office. This is the story of where the agent shined, where it quietly lied to me, and the exact line I now draw between what it runs and what I sign.

The setup: stop asking, start building

My first instinct was the obvious one. Paste a CSV into a chat and say "categorize these." I did that for about ten rows before I stopped, because it was the wrong shape of work.

A throwaway categorization is a thing you ask for once and then have to babysit forever. What I actually wanted was a process I could re-run next month with a different CSV and trust a little more each time. So I told Claude Code to build the bookkeeping pipeline, not to do the bookkeeping.

That distinction matters more than it sounds. When you ask an agent to build the thing that does the work, you get a script you can read, a set of rules you can correct, and an audit trail you can point at later. When you ask it to do the work, you get an answer and a shrug. One is an asset. The other is a chore you now share with a robot.

The pipeline it wrote was unglamorous and exactly right:

Pull the month's transactions from my accounting platform over an MCP connection.
Match each bank row to a receipt or invoice by amount and date.
Propose a category for every row, with a one-line reason.
Flag anything it was not sure about instead of guessing silently.

Step 4 is the one that saved me. More on that in a second.

Where the agent earned its keep

The reconciliation itself was genuinely good. Matching 200 bank rows against receipts is the kind of tedious pattern-matching that humans are bad at precisely because it is boring. You zone out around row 40 and start rubber-stamping. The agent does not zone out.

It correctly handled the cases I expected to trip it: a subscription that renewed on a slightly different day, a refund that showed up as a negative line, a vendor whose name on the bank statement bore no resemblance to the name on the invoice. For the bulk of the month, "AWS-style charge goes to infrastructure, coffee receipt goes to meetings" was handled without me touching anything.

This is not a niche experience anymore. A January 2026 Deloitte study found 63% of finance organizations have fully deployed AI somewhere in their operations, and the pattern that keeps winning is the boring one: let the model categorize at volume, then have a human review the output. Machines do the reading, people do the signing. I arrived at the same split independently, which I choose to read as validation rather than as me being unoriginal.

The 11 it got wrong

Here is the uncomfortable part. The 11 mistakes were not random noise. They clustered in three places, and all three are places where the agent had no way to know what it did not know.

Intent it couldn't see (5 rows). A laptop I bought is a business expense if I use it for work and a personal purchase if I do not. The receipt looks identical either way. The agent categorized every device as a business asset because that is the statistically likely call, and for two of them it was wrong. No amount of context in the CSV would have told it otherwise. That information lives in my head.

Rules it didn't have (4 rows). Tax categories are not universal logic; they are local law. A meal with a client and a meal alone are deductible to different degrees depending on rules the agent was never given. It made a reasonable, confident, wrong guess. Confidence is the dangerous part. A wrong answer delivered with a hedge is easy to catch. A wrong answer delivered cleanly slides right through.

Edge cases that needed a human (2 rows). A single payment that covered two unrelated things, split across categories. The agent picked one. A person would have asked.

Notice what is not on this list: arithmetic. It never added wrong, never lost a row, never double-counted. The failures were all judgment, not math. Which is the whole point. The agent is a tireless clerk, not an accountant, and the moment I treated it like an accountant was the moment it would have cost me money.

The line I now draw

After this month I have a rule, and it is not "trust the agent" or "don't trust the agent." It is narrower than that.

The agent runs everything that is high-volume and low-judgment: pulling data, matching receipts, drafting categories, flagging doubt. I personally sign off on everything that is low-volume and high-stakes: every flagged row, a spot-check of the confident ones, and every category that touches a tax outcome.

The flagging mechanism is what makes this tractable. Because I told the agent to surface its own uncertainty rather than bury it, my review was not "re-check 200 rows." It was "check the 18 it wasn't sure about, then sample the rest." Seven of the 11 errors were already sitting in its own flagged pile. The other four I caught on the sample. That is the difference between an agent that helps and an agent that just moves the work somewhere you can't see it.

One thing I will say loudly, because the internet is full of people who won't: the screenshot-friendly version of this post would be "AI did my taxes in an afternoon." That version is a lie of omission. The honest version is "AI did 94.5% of my taxes and I did the 5.5% that could get me audited." The second one is less viral and considerably more useful.

What I'd tell you to copy

If you want to try this on your own books, three things carried the result:

Build the pipeline, don't request the answer. A script you can read and correct beats a chat reply you have to trust. Next month you re-run it; you don't re-explain it.
Make uncertainty a first-class output. An agent that flags its own doubt turns a 200-row audit into an 18-row one. Telling it to never guess silently was the single most useful instruction I gave.
Keep judgment on your side of the line. Volume to the machine, stakes to the human. Intent, local law, and genuine edge cases are not data problems you can prompt your way out of.

I spent years learning that the bottleneck in any process is rarely the part everyone optimizes. With bookkeeping, everyone wants to automate the data entry. The data entry was never the hard part. The hard part is the eleven rows that need a human who knows what the business actually did, and an agent that is honest enough to point at them.

I write more about this kind of human-and-agent division of labor in Claude Code Mastery, where a full chapter goes into using coding agents for financial and business work without handing over the parts that bite.

Next month the agent runs the books again. I will still read every flagged row. Let's keep it interesting.

AI Citations Have a Half-Life. I Tracked Mine for 9 Weeks and Watched Them Decay.

Ken Imoto — Tue, 23 Jun 2026 13:00:01 +0000

I have written about measuring AI citations more times than I have written about my own family, which says something I would rather not examine. KPIs, tracker comparisons, how broad my citation footprint is. All of it was a snapshot: how many engines cite me today. It took me an embarrassingly long time to ask the obvious follow-up. Not "how many," but "how long."

So I started logging citations weekly instead of once. Same prompts, same five engines, every Monday morning, for nine weeks. The result is the thing nobody screenshots on LinkedIn: a citation is not a trophy you win and keep. It is a perishable good. Mine peaked around week three and then lost more than half its value, and the strangest part is that my Google search traffic for the exact same pages did not budge.

What I actually measured

I had three articles that consistently got cited across ChatGPT, Perplexity, Gemini, Claude, and Brave AI. I picked those three because a citation that only fires once is noise, and I wanted signal I could track over time.

Every Monday I ran the same 30 prompts, three times each, and logged how many returned one of my three pages as a clickable citation. I also pulled the weekly Google Search Console clicks for those exact URLs, so I had two curves drawn on the same calendar.

One curve is AI citations. The other is plain old search traffic. I expected them to roughly track each other. They did not even rhyme.

The decay curve

Here is what nine weeks looked like, normalized so the peak week is 100.

Week:        1    2    3    4    5    6    7    8    9
AI cites:   48   82  100   91   70   54   47   44   46
GSC clicks: 88   92   95  100   97   99   96   98   97

The AI citation line climbs for three weeks, tops out, and then slides down a ramp until it settles at roughly half its peak. The Google line is a heartbeat monitor on a calm patient: it wanders inside a narrow band and never does anything dramatic.

If I fit a half-life to the decay portion, my citations lost 50% of their peak rate in about four to five weeks. That lined up unnervingly well with what other people are measuring out loud right now. One platform-by-platform study put the median AI citation half-life at 4.5 weeks, with ChatGPT churning fastest at about 3.4 weeks and Perplexity holding longest near 5.8. Another found that AI-cited domains turn over 40 to 60 percent every month. I was not looking at a quirk of my tiny blog. I was looking at the weather.

Why the two clocks disagree

The reason the curves diverge is that they are pulling from different places.

Google's organic ranking for an established page is sticky. The page earned its position over months, the ranking signals are slow-moving, and one week of nothing-in-particular does not dislodge it. That is the heartbeat line.

AI citations are pulled from a live index at answer time, and that index is biased toward fresh material. Industry measurements in 2026 keep landing on the same shape: roughly half of all AI-cited content is under 13 weeks old, and pages updated within the last 30 days earn several times more citations than older ones. One freshness study even found ChatGPT cites pages that are, on average, over a year newer than the URLs Google ranks organically for the same query. My three pages were great answers in March. By May they were still great answers, but the engine had three newer "great answers" to choose from, and it is graded on recency, not on loyalty to me.

So the decay is not a quality problem. My pages did not get worse. The pool they compete in got younger.

The part where I tried to fight it

Naturally, my first instinct was to refresh the pages and reset the clock. The freshness research says a substantive update (a new stat, a corrected claim, a visible timestamp change) is enough to re-trigger the freshness signal. So in week seven I rewrote the intro of my best-performing page, added a 2026 data point, and bumped the dateline.

You can see the result in the table. Week eight ticked from 47 to 44, then week nine recovered to 46. Within the noise. A genuine nothing.

I want to be honest about that, because the tidy version of this post would have the refresh save the day in a triumphant week-nine spike. It didn't. One refresh of one page over two weeks is not enough signal to conclude the tactic failed, but it is enough to stop me from promising you it works. What I changed was small, and the engines treated it as small. The lesson I am taking is that a cosmetic timestamp bump is not a refresh. The studies that show refreshes working describe substantive rewrites, and "I added one sentence" is the SEO equivalent of changing your shirt and calling it a new wardrobe.

What this means if you write about LLMO

Three things rearranged in my head.

A citation count is a flow, not a stock. I had been treating "I am cited by five engines" like money in a bank account. It is closer to water in a leaky bucket. If you measure once and frame it, you are reading the level at one instant and assuming it holds. It does not hold. The only honest version of the metric is a rate over time, which means you have to measure on a cadence or you are measuring nothing.

The refresh cadence is the actual job. If half your citation value evaporates in roughly a month, then the maintenance schedule matters more than the next new post. I spent six months optimizing the title and schema of pages at the moment of publication, which is the one moment the freshness clock is on my side anyway. The hard part is week six, when the clock has turned against me and the only move is real work on an old page. And the cadence is not one-size-fits-all: the platform studies suggest refreshing ChatGPT-facing content roughly biweekly while Perplexity tolerates a six-week cycle, which means a single refresh calendar is already a compromise.

SEO and AEO are not the same investment. They feel adjacent because they both start with a URL and a query. But my data shows them moving on different timescales for the same pages, which means a single content calendar optimized for one is probably mistimed for the other. Search rewards the durable asset. AI citation rewards the recently-touched asset. Those are not the same content strategy wearing two hats.

For the framework side of this, where the goal is a citation-maintenance loop instead of a one-time launch checklist, the LLMO Framework splits its pillars into Citability (do you get cited at all) and the Authority and Coherence signals (do you stay cited). Its refresh-strategy guidance is the part I had ignored. Nine weeks of staring at a decay curve made me realize I had spent all my effort on the first pillar and almost none on the rest. Getting cited is a launch problem. Staying cited is a retention problem, and retention is where the freshness clock lives.

What I'm doing now

I turned the weekly log into a standing job instead of a one-time experiment. Every Monday the same 30 prompts run, the same three URLs get checked, and the number goes into a spreadsheet next to the GSC clicks. When a page's citation rate drops below half its peak, it goes on a refresh list, and "refresh" now means a real new section or a new dataset, not a dateline nudge.

I am also going to stop reporting my citation footprint as a single proud number. "Five engines cite me" is true on a Monday and a lie by the next month. The number I trust now is the trend line, and the trend line only exists if you keep measuring after the launch-week dopamine wears off.

The blog never stopped being a measurement target. I just learned that the measurement has a time axis, and the time axis is where all the interesting bad news lives.

I will re-run the full nine-week log on a different set of pages next quarter to see if the half-life holds at four-to-five weeks or if it varies by topic. My guess is that evergreen how-to pages decay slower than my "here's what broke this week" experience reports, because the how-to answers a stable question and the experience report answers a question nobody will type again. We'll see. The nice thing about a leaky bucket is that it makes a very honest dashboard.

If you want the measurement loop I keep referring to — the five-prompt, three-retry, monthly-cadence setup with the GA4 segment regex and the Python visibility script — I wrote it up in Optimizing for AI Search (LLMO). This post is what happened when I added a time axis to that loop and watched the numbers fall.

I Gave My Agent Persistent Memory. It Remembered the Wrong 3 Things for a Week.

Ken Imoto — Mon, 22 Jun 2026 13:00:00 +0000

I gave my coding agent persistent memory in March. By the end of the week it was telling me, with total confidence, that I preferred Poetry for dependency management, that our staging database lived in us-east-1, and that I had already approved a migration plan I had explicitly rejected two days earlier.

All three were wrong. I use uv. Staging moved to ap-northeast-1 back in February. And I never approved that plan.

The annoying part is that none of these were hallucinations in the usual sense. The agent wasn't making things up on the spot. It was faithfully recalling facts it had written down about me at some point and never corrected. The memory worked exactly as designed. That was the problem.

The pitch everyone believes

The standard story about agent memory goes like this. Every session starts with a blank context window. The agent forgets you the moment the conversation ends. So you give it persistent memory: a place to write down your preferences, your project state, the decisions you've made. Next session, it reads those notes first and picks up where it left off.

Anthropic shipped exactly this for all Claude users in March 2026. Claude now scans your history, synthesizes a summary of who you are and what you're working on, and refreshes it roughly every 24 hours. Letta (the framework formerly known as MemGPT) goes further: the agent edits its own memory blocks through tool calls, deciding what's worth keeping.

The selling point is continuity. No more re-explaining your stack every morning. And for the first few days, it genuinely felt like working with someone who remembered me.

Then it started remembering things that were no longer true.

Stale memory is worse than no memory

Here's the uncomfortable asymmetry. An agent with no memory asks you the same question every day. That's annoying, but it's honest. You answer, it acts, nothing rots.

An agent with persistent memory answers the question itself, using a fact it learned three weeks ago. If that fact has changed, you don't get a question. You get a confident wrong action. And because the answer sounds informed, you're less likely to catch it.

My us-east-1 bug is the clearest example. The agent had recorded our staging region back when it was true. We migrated. Nobody told the memory. So for a week the agent kept generating deploy commands pointed at a region with nothing in it, and every command looked perfectly reasonable because the region string was a real region we had genuinely used.

This is the part the memory pitch skips. "Remembering" and "remembering correctly" are different features, and persistence only gives you the first one for free.

The three ways memory goes bad

After staring at my agent's memory file for an embarrassingly long evening, I sorted the failures into three buckets. They're not exotic. They show up the moment a memory system runs longer than a few days.

Stale facts. Something that was true when it was written and isn't anymore. Regions, versions, deadlines, who owns which service. The world moves; the note doesn't. This was most of my pain.

Poisoned facts. A wrong fact gets written once and then quoted forever. The "I approved the migration" entry came from a single ambiguous message where I said "yeah that approach makes sense" about the general shape of a plan. The agent compressed that into approval, wrote it down, and from then on treated it as settled history. No amount of arguing in later sessions dislodged it, because it kept reloading the poisoned note at the start of each one.

Over-confident summaries. The Poetry thing was this. I'd mentioned Poetry once, months ago, in the context of a different repo. The summarization pass that builds the daily profile flattened "used Poetry on one old project" into "prefers Poetry." Summaries are lossy by design, and the loss tends to drift toward overconfident generalizations.

The first one is a freshness problem. The second is an integrity problem. The third is a compression problem. Lumping them together as "the agent got confused" is exactly why they're hard to fix.

Why self-editing memory doesn't save you

The obvious response is: let the agent manage its own memory. It edits, it prunes, it corrects. Letta is built on this idea, and it's a genuinely good idea.

But teams moving Letta from prototype to production keep hitting the same wall, and I hit it too: self-editing memory is unpredictable. The agent decides what to keep based on the same flawed judgment that wrote the bad fact in the first place. When I corrected the region in a session, the agent sometimes updated the memory block, sometimes wrote a second entry that contradicted the first, and once helpfully "consolidated" both into a summary that kept the wrong one. Letta even shipped a Recovery-Bench benchmark in 2026 specifically to measure how well agents climb out of corrupted states, which tells you the industry knows this is real.

The deeper issue: an agent editing its own memory has no external source of truth to check against. It's grading its own homework. If it believed us-east-1 yesterday, "us-east-1" looks consistent with everything it knows today.

I learned this lesson once before, the hard way, with a junior engineer I onboarded years ago. Brilliant, fast, and absolutely certain about a deployment process he'd learned on his first day. The process had changed in month two. He kept doing it the old way for weeks, confidently, because nobody handed him a reason to doubt his own notes. Persistent memory gave my agent the exact same failure mode, minus the part where a human eventually overhears the mistake at lunch.

What actually started working

I'm not going to pretend I solved this. But three changes cut the wrong-fact rate to something I can live with.

Timestamp everything, and decay it. Every memory entry now carries when it was written. Facts about volatile things (regions, versions, deadlines) get treated as suspect after a set window and re-confirmed rather than trusted. A region string from three weeks ago isn't a fact; it's a hypothesis with an expiry date.

Separate "observed" from "inferred." The Poetry disaster came from the agent storing a generalization as if it were a stated preference. Now there's a hard line: things I literally said go in one bucket, things the agent concluded go in another, and the inferred bucket needs more evidence before it gets to drive an action. Augment's framing stuck with me here: memory should guide decisions, but never be treated as infallible truth.

Make corrections destructive. When I correct a fact, the old entry doesn't get a polite contradicting neighbor. It gets overwritten and logged. The audit log matters more than I expected. The first time the agent confidently cited a fact I didn't recognize, being able to see when and from which message it was written turned a mystery into a one-line fix.

None of this is exotic. It's basically the discipline you'd apply to any cache: TTLs, write provenance, explicit invalidation. We just forgot to apply it to memory because the word "memory" makes it sound like something more trustworthy than a cache. It isn't. It's a cache that talks back.

The uncomfortable takeaway

Persistent memory doesn't make an agent reliable. It makes an agent consistent, which is a different thing, and occasionally the opposite one. A consistent agent repeats yesterday's truth whether or not it's still true today.

The fix isn't more memory. It's treating every remembered fact as a claim with a timestamp and a source, not as ground truth. The agents that stay useful over weeks aren't the ones that remember the most. They're the ones that know which of their memories to distrust.

If you want to go deeper on how context, state, and memory actually interact in production agents, I wrote about the full picture in Context Engineering.

My agent still remembers the wrong region sometimes. But now it asks before it deploys there. That's the whole game.

I Stopped Adding Context to My Agent and Pruned Tool Outputs Instead — My 3-Hour Task Stopped Forgetting Its Own Plan

Ken Imoto — Fri, 19 Jun 2026 13:00:01 +0000

For a long time I treated context like savings: the more I put in, the richer I'd be. Thick CLAUDE.md, every file that might be relevant, the full output of every tool left sitting in the window. More information, smarter agent. That was the theory.

The theory fell apart three hours into a migration task. The agent had set itself a design rule in the first twenty minutes: don't touch the legacy adapters, wrap them. By hour three it had forgotten its own rule and edited two of them directly. It also wandered into a directory I had explicitly told it to leave alone. The prompt wasn't the problem. The context had gotten so fat that the one instruction that mattered was buried under everything else I'd helpfully shoveled in.

So I did the opposite of my instinct. I stopped adding and started pruning. Tokens dropped from about 140K to about 84K, roughly 40%, and the long task got more accurate, not less. This is the story of what I cut.

The point where "more is smarter" turns into a lie

Context has a ceiling on how much of it actually works, and the ceiling sits well below the advertised number.

Claude Sonnet markets a 200K-token window. But Sourcegraph's Geoffrey Huntley reported quality starting to slide somewhere around 147,000–152,000 tokens, what he calls redlining. The capacity of the window and the capacity you can use are two different numbers.

This is not my anecdote talking. Chroma's research team ran the experiment properly: they tested 18 frontier models on how rising input length affects output quality, and every one degraded as the context grew. They named it context rot. A model with a 200K window can show measurable degradation long before it's full. And the kicker: how you fill the window matters. Padding it with tool operations that partly cancel each other out hurt performance more than padding it with neutral text. Raw tool dumps are close to the worst-case filler.

Picture a new hire. Hand them three pages and they're useful by lunch. Bury the same desk under three hundred pages and they'll spend the day just figuring out which page matters. Information and usefulness stop being friends at some point on that curve. My agent had hit that point, and I was the one stacking the pages.

The three things I pruned

When I went looking for what to cut, it sorted into three buckets.

1. Raw tool outputs

This was the big one. The full log of npm test. The four hundred lines grep returned. The giant JSON body from an API call. The agent hoards all of it, verbatim. But the only thing that moves the next step forward is the conclusion: "three tests failed, here are the files." The rest is ballast.

Anthropic now ships this as an actual feature, which told me I hadn't invented anything; I'd just been doing it by hand. Their context editing clears old tool results past a token threshold and leaves a small placeholder so the model knows something was removed. In a 100-turn web-search eval, Anthropic measured context editing cutting token use by 84% while keeping workflows alive that would otherwise have run out of room. As of mid-2026 the whole platform leans this way: dynamic tool filtering and programmatic tool calling all chase the same goal of keeping the window lean. The mechanism I'd been hacking together with notes-on-the-side had a name and a measured number.

2. Irrelevant files

The "let me read this just in case" files. On a migration task I'd opened five components that had nothing to do with the migration. I'd told myself it was insurance. It was noise I paid for in tokens.

3. Stale conversation turns

The early flailing. Once "we're going with approach B" is decided, the three rejected approaches that got us there are dead weight. Keep the decision, drop the path to it. A /compact with a custom instruction does this without throwing away the parts you need.

The numbers, before and after

Same migration task, run with the fat context and then with the pruned one.

Metric	Before pruning	After pruning
Tokens used	~140K	~84K
Design rule held?	Drifted near the end	Held to the end
Times I had to re-instruct	6	1
Wrong files touched	2	0

Tokens fell about 40%. But the number I actually cared about was the re-instruction count going from six to one. The agent kept its own hour-one decision all the way to the finish because it never climbed into the 147K–152K redline where the rot sets in. I didn't make it smarter. I stopped making it dumber.

And notice the direction here. A while back I ran the opposite experiment, stacking four more context layers on top of RAG and measuring the gain. That was about adding structure and watching the curve rise (until it fell over on the smaller model). This is the mirror image: removing noise and watching accuracy come back. Same window, opposite vector. Both experiments point at the same uncomfortable truth: the window is not a bucket you should try to fill.

Why "don't put it in" is harder than "put it in"

Honestly, pruning is the harder discipline.

Adding is free of judgment. Nervous about a file? Open it. No decision required. Pruning forces you to say "this isn't needed" and then sit with the fear that it was. Every cut is a small bet against your own anxiety.

My rule for the bet is one question: does this directly help the single step in front of the agent right now? If not, it stays out. If it turns out I was wrong, the agent can go read the file again; it's an agent, retrieval is its job. Pre-loading everything wasn't serving the model. It was sedating me.

Anthropic's newer models track how much of their own context is left, a kind of context self-awareness, so they can pace a long task instead of sprinting into the wall. But that only helps if there's headroom to track. Fill the window with raw logs on turn one and there's no runway left to be aware of.

Takeaway

When a long task started losing accuracy, my first instinct was "it doesn't have enough information." Exactly backwards. It had too much, and the one instruction that mattered had been diluted to nothing.

What I actually did was three cuts: replace raw tool output with its conclusion, stop opening files "just in case," and throw away the trial-and-error once a decision is made. Forty percent fewer tokens, no trip into the rot valley, and a plan that survived three hours intact.

Context engineering sounds like a question of what to add and how to arrange it. On a long-running task, the move that paid off was the other one: deciding what never goes in. Clear the desk down to three pages. That's the moment the new hire becomes useful again.

The full map of context design (how System Prompt, few-shot, and RAG fit together, and where adding more crosses the 80-20 line into actively hurting you) is in my Context Engineering Practical Guide. This post is the subtraction side of that book, stress-tested on a task long enough to make context rot show up.