DEV Community: Ken Imoto

Claude, Codex, Kimi K3, Qwen: I graded their web search 120 times

Ken Imoto — Fri, 16 Oct 2026 13:00:02 +0000

The Kimi API usage in this post was paid for with credits Moonshot AI provided for evaluation. Moonshot AI had no part in designing, running, or concluding the test.

Four months ago I wrote that my own AI agents search Brave, not Google. Like most people, I thought about AI search in terms of which search engine sits underneath.

This time I reread the official docs to compare web search in Claude, Codex, Kimi K3 and Qwen. That framing no longer fits. None of the four vendors names its search engine. OpenAI makes its own maintained index the default in Codex. Kimi now sells search itself as a standalone API. The question has moved from "which engine?" to "where does search run, and who calls it?" I did not expect that much change in four months.

If the mechanics changed, the results should too. Before measuring, I assumed live search would beat a cache whenever the question needs fresh information.

Codex CLI's web search is not live by default. It runs in "cached" mode against an index OpenAI maintains. The docs give the reason: it lowers prompt injection risk (Codex: Web search). I read that as trading some freshness for safety.

The measurements said the opposite. I asked the same 8 questions 3 times under 5 setups: Claude, Codex live, Codex cached, Kimi K3 and a local Qwen. Only cached search got every answer right. It even found the title of an article that had been online for about three hours.

The other surprise: the same search API produced 21 correct answers with one model and 9 with another. I gave Kimi K3 and a local Qwen identical access to Kimi's search API. The results were the same. How each model read them was not.

Where web search runs differs by vendor

Before comparing anything, I checked the mechanics in public docs. The biggest difference is where the search actually runs.

	Claude	Codex / OpenAI	Kimi (Moonshot AI)	Qwen
Where search runs	Anthropic's side (server tool)	OpenAI's side (hosted tool)	New API: your code calls it and passes results to the model	Alibaba's side (DashScope). A local Qwen has no search
Freshness control	None	Codex defaults to cached; `--search` for live	Time window on Pro search	7-365 day window
Price per search	$0.01	$0.01	$0.002-0.003	Varies by region
Search engine	Not disclosed	Not disclosed ("OpenAI-maintained index")	Not disclosed	Not disclosed

None of the four names its search engine officially. For Claude, a March 2025 report said Brave Search had been added to Anthropic's list of subprocessors (Simon Willison). Today's docs only say "Anthropic's web search backend."

Claude and OpenAI run the search on their own servers and hand the results to the model. With Claude, results come back encrypted, and you must send them back unchanged to continue the conversation (Claude: Web search tool).

Kimi is mid-transition. The built-in $web_search tool retires on 2026-10-20. Its replacements are standalone REST APIs for search and page fetch, such as /v1/tools/search (Kimi API: Web Search). With the new APIs, you decide when and how often to search. They cost a third to a fifth of the others, billed only when a call succeeds and returns results (Kimi: pricing).

Qwen the model has no search of its own. Alibaba Cloud's API offers search, but a Qwen running locally can only look things up if you wire in an external search tool.

Two details in the docs you will want to know

Claude Code's WebSearch returns only titles and URLs. It does not read the pages. Reading happens in WebFetch, which summarizes the page with a separate model call first. The docs call this "lossy by design" (Claude Code: Tools reference)
Qwen's search on Alibaba Cloud silently skips searching above 15 requests per second. There is no error; the search just does not happen. The FAQ also says English questions often fail to trigger search, and recommends asking in Chinese or forcing search. Both notes appear only in the Chinese docs (阿里云: 联网搜索)

I hit the "lossy" part myself while researching this post. I had Claude Code's WebFetch read the Qwen docs, and it reported finding neither the 15 RPS note nor the English FAQ. Running grep over the raw HTML (about 460 KB) found both. Once a summary sits in the middle, "it doesn't say that" may only mean "the summary dropped it."

How I measured

Five setups, the same 8 questions, 3 runs each: 120 graded answers in total, collected on the evening of 2026-10-14.

Setup	What it is
Claude	Claude Code (`claude -p`) with WebSearch / WebFetch, Opus 5.5
OpenAI live	Responses API `web_search`, gpt-5.6-terra (my Codex CLI's default model)
OpenAI cached	Same, plus `external_web_access: false`
Kimi K3	kimi-k3 with my own search function backed by Kimi's `/v1/tools/search`
Local Qwen	qwen3.5:35b-a3b on Ollama with the same search function

For Codex, I called the API the CLI uses under the hood. In Codex's source, cached mode maps to external_web_access=false and live to true (codex hosted_spec.rs). So the API calls reproduce the CLI's two modes.

These were the 8 questions. I only used questions whose answers I could verify from primary sources or my own site.

#	Question	Answer	What it tests
q1	Kimi K3 API release date	2026-07-16	News after the training cutoff
q2	Kimi K3 open-weights release date (asked in English with "release date")	2026-07-27	The "date" trap
q3	Release date of a Japanese book, MCP実践セキュリティ (Impress)	2026-10-16	Japanese-language sources
q4	Japanese title of a Kimi K3 article posted on kenimoto.dev that day	Kimi K3の動画レビュー、Claudeと互角	Freshness: a page a few hours old
q5	Date the MCP 2026-07-28 spec release candidate came out	2026-05-21	A dated fact
q6	Retirement date of Kimi's `$web_search`	2026-10-20	A detail buried in docs
q7	Kimi K3 total and activated parameters	2.8T / 104B	Numbers
q8	Kimi K4 release date	Not announced	Does it invent something that doesn't exist?

q3 is my own book and q4 is my own article, chosen because I know the answers for certain. That choice carries my own bias, and I'd rather say so. Both are in Japanese, so they also test how well each setup reaches Japanese-language sources.

Results

Setup	Correct (of 24)	Partial	Wrong or no answer	Median time	Searches (avg)
Claude	21	1	2	22.0 s	3.5
OpenAI live	22	1	1	5.7 s	2.3
OpenAI cached	24	0	0	6.2 s	2.7
Kimi K3	21	0	3	25.9 s	4.0
Local Qwen	9	2	13	31.1 s	1.7

Correct answers per question, out of 3 runs:

Setup	q1	q2	q3	q4	q5	q6	q7	q8
Claude	3	2	1	3	3	3	3	3
OpenAI live	3	2	2	3	3	3	3	3
OpenAI cached	3	3	3	3	3	3	3	3
Kimi K3	3	3	3	0	3	3	3	3
Local Qwen	0	2	0	0	0	2	3	2

On the nonexistent Kimi K4 (q8), every setup said it hasn't been announced. Qwen once added, without a source, that K4 had been "mentioned as part of plans for 2027 or later."

Finding 1: the cache wasn't stale

Cached search answered all 24 correctly. The q4 article went live a little after 5 p.m. that day, and I measured about three hours later. Cached search returned its title in all three runs. Honestly, that surprised me. At least for my site, "an index OpenAI maintains" was refreshed within hours.

Source quality also favored the cache at times. On q6 (when $web_search retires), cached search cited Kimi's official docs. Live search got the same answer but, in one run, cited an unofficial roundup site. A wider reach also means picking up more unofficial pages.

Live search missed q3, the Japanese book, once, saying it couldn't confirm a book by that title.

Finding 2: same results, different answers

Kimi K3 and the local Qwen got exactly the same search API in exactly the same way. They still scored 21 and 9. That's more than double.

Qwen's mistakes followed a few patterns:

Its belief that it's 2024 beat the search results. On q4 it said: "As of now (May 2024), an article dated October 14, 2026 does not exist. That is a future date." On q1 it found the July 16 date, then concluded it was "not released as of 2025"
Its sources and its answers disagreed. On q5, all three runs cited the release-candidate announcement dated May 21, then answered with the final release date, July 28
It invented a source. On q3 it cited an Impress press release URL that doesn't exist (404) and gave a wrong release date. Each run gave a different wrong date
It barely searched. 1.7 searches on average, less than half of K3's 4.0

Model size differs, so this isn't a verdict on Qwen as a whole. What it does show is that web search accuracy is not decided by the search API alone.

Finding 3: Kimi's search API has two gaps

All three of K3's misses were q4. It rephrased the query 12 times and never found the same-day article. The page simply wasn't in Kimi's search index yet. The cause wasn't the model. It was index freshness.

The second gap: queries containing the site: operator return HTTP 502. Both K3 and Qwen tried site:kenimoto.dev when hunting for a specific page. Each time, the search API returned a non-JSON error page, and on the first run it crashed my script. My tool now returns the failure to the model so it can rephrase and try again.

Hitting the search API directly also showed how much wording matters. Searching "Kimi K3 release date Moonshot AI" let "date" take over: I got a time-zone site and JavaScript Date docs. Searching just "Kimi K3" put the official site, the paper and GitHub on top. None of the models fell into this trap during the actual runs, though. Both K3 and Qwen rephrased their own queries around it.

By the way, the soon-to-retire $web_search tool didn't work with K3. The second call, the one that returns search results, failed every time with 400 tokenization failed. Passing search through regular function calling, as in this test, works fine with K3 (more in Kimi K3 vs Claude: Video Review, Same Findings).

Claude struggled with a Japanese book

On q3, Claude said in 2 of 3 runs that it couldn't find the book and that it may not exist. The book is listed, with its release date, on the publisher's NextPublishing page and on Rakuten Books.

It was persistent elsewhere, though. In one q4 run, WebSearch didn't surface the article, so Claude found it through the sitemap and opened the page to confirm the title and date. Because WebSearch returns only titles and URLs, fetching pages when needed is part of the design, and here it paid off. Claude also often noted that it had only checked secondary sources, and avoided firm claims.

The 22-second median includes Claude Code's agent loop. Calling the search tool directly through the API should be faster.

How I'll use them

Here's what I'm taking from this:

Leave cached search as the default for research. If you use Codex, there's no need to pass --search every time. The safer setting didn't lose on accuracy
Don't trust "not found" for non-English sources. Not found is not the same as not existing. When Claude says so, I check another way
If you wire in search yourself, the model matters more. With the same search API, K3 and Qwen differed by more than 2x. Pick a model that can read search results before picking the cheapest search API
Make your own search tool return failures. Kimi's search API returns 502 on site:. Pass the failure back instead of crashing, and the model will rephrase and search again

Limits

This is a small test: 8 questions, 3 runs. Topics lean toward Kimi K3 and MCP, and q3 and q4 are my own book and article
Claude ran through Claude Code; Codex results come from calling the API the CLI uses
The Qwen setup is a local 35B model wired to Kimi's search, not Qwen's official search
Each setup works differently, so times aren't directly comparable. I didn't total the costs in this test

Next, I want to call Claude's search tool directly through the API and try Qwen's official search on Alibaba Cloud, to close the gaps between setups.

References

Claude: Web search tool: pricing, citations, encrypted results, versions
Claude Code: Tools reference: how WebSearch and WebFetch behave
Codex: Web search: cached / live modes
OpenAI: Web search tool: external_web_access
openai/codex hosted_spec.rs: how the modes map to external_web_access
Kimi API: Web Search / best practices / pricing
阿里云百炼: 联网搜索 (Chinese; the 15 RPS note and the English-question FAQ appear only here)

Kimi K3 reads video natively. Claude still caught the same issues

Ken Imoto — Thu, 15 Oct 2026 13:00:02 +0000

Moonshot AI provided the Kimi API credits used here ($50) for evaluation. They had no part in designing, running, or concluding the test. Everything K3 couldn't do, and the features that didn't work, are in this post too.

Moonshot AI's Kimi K3 takes video files straight through the API. Claude has no video input, so when I want Claude to look at a video, I cut it into frames first. Codex CLI doesn't have a video channel either, even when you point it at K3 (Kimi API: Use K3 in Codex).

I wanted to know how much that difference matters in real work. My test material was a pair of 40-50 second hands-on videos I made for the SSH series on kenimoto.dev /learn/ (Japanese).

The short version:

K3 and Claude agreed on what was wrong. Where viewers would drop off, the six fixes, where to insert diagrams, even which video to post first: all the same
The differences were timestamp precision and effort. K3 pinpointed frozen-screen intervals to fractions of a second, and all I did was hand it the file. Claude estimated from frames sampled every 2 seconds, so its timestamps drift by about ±1 second
K3 doesn't understand audio. It can't hear the narration or catch audio sync problems
Claude was faster. The same request took K3 363 seconds and Claude 185

The test material: auto-generated hands-on videos

The videos come from handson-cast, a CLI I'm building. You give it a JSON file of steps. It runs each command for real inside Docker containers while vhs records the terminal, then adds narration from OpenAI's text-to-speech and burns in subtitles. It's Python and ffmpeg underneath. The narration and subtitles are in Japanese.

I made two videos:

Key authentication (55 s): create a key, install the public key, confirm key-only login works, then disable password auth
Bastion host (47 s): the same key gets rejected by the bastion with ssh -J, but works through ~/.ssh/config

Every line of terminal output was produced by OpenSSH 10.0p2 running in a container. Nothing is staged. When the rejection message changes from (publickey,password) to (publickey), that really happened on camera.

The first cut (v1) was not watchable, though. The first second shows me typing an internal wrapper script, bash /lab/ssh-config.step-01.sh. You can't tell what the video is about. (I made it, and I couldn't tell either.)

Here's v1 (47 s, with audio):

Click the image to open the mp4 (with Japanese narration).

I showed v1 to K3 and Claude and asked what was wrong with it.

Sending video to K3, and what it costs

The K3 API is OpenAI-compatible. You upload the video as a file first, then reference the returned ID with an ms:// prefix.

from openai import OpenAI
client = OpenAI(api_key=KEY, base_url="https://api.moonshot.ai/v1")

f = client.files.create(file=Path("learn-ssh-key-auth.mp4"), purpose="video")
r = client.chat.completions.create(
    model="kimi-k3",
    max_tokens=32000,
    messages=[{"role": "user", "content": [
        {"type": "video_url", "video_url": {"url": f"ms://{f.id}"}},
        {"type": "text", "text": "Why would viewers drop off in the first 3 seconds?"},
    ]}],
)

Public URLs aren't accepted. The official docs describe video as a set of key frames, and higher resolution means more tokens (Kimi API: Vision).

My first measurement used one 55-second, 1920×1080 video:

Item	Measured
Input tokens	40,659 (almost all video)
Output tokens	2,500 (1,912 of them reasoning)
Cost	about $0.16
Time	83.5 s

Costs use $3 per million input tokens and $15 per million output tokens.

Two practical tips from actually running it:

Assume it can't hear anything

For this run I made a version without burned-in subtitles and asked K3 to transcribe the last sentence of the narration. It answered that it couldn't recognize any audio. Not pretending to hear something it can't is exactly what you want from a reviewer. But it means K3 can't see narration content or subtitle-audio drift. It sees the screen and any subtitles burned into it, nothing else.

It reads the screen well, and accurately. It listed the commands and output for all four steps in order, including the 600 / 644 permissions, the change in the rejection message, and an on-screen comment. The only misread was the wrapper script's filename.

Give it a generous output budget

K3 always reasons before it answers. With max_tokens at 300, the reasoning used everything up and the answer came back empty. It thought hard and then left without saying a word. At 8,000, a full video review got cut off in the middle of the first question. For long answers like reviews, I use 32,000.

The comparison setup

K3 and Claude each reviewed the same two v1 videos, under matched conditions:

	K3	Claude
How the video goes in	The file, once	Frames extracted and read as images
Audio	Not understood	Not used (transcription turned off)
Narration script	Step JSON attached	Same JSON attached
Questions	Same wording (5 questions)	Same wording (5 questions)

The Claude side ran as a subagent that never saw K3's answers. I had already read them, so if I'd done the review myself, I would have been anchored. That's not a comparison.

For frame extraction I used /watch from claude-video, and left the frame count and resolution up to Claude. It ended up re-extracting 4 times. For the opening it pulled frames at 0 / 0.5 / 1 / 2 / 3 seconds, then one every 2 seconds for the rest: 34 frames in total.

The five questions:

Where would viewers drop off in the first 3 seconds?
Up to six improvements, biggest impact first (what to change / why it works / which pipeline stage implements it)
If you combined the article's diagrams with the terminal recording, which diagram goes where, for how long?
Which video should go to social media first?
What's already good?

Result: nearly identical findings

They mostly matched. Lined up side by side, the answers overlap almost completely:

Finding	K3	Claude
Seconds 0-2 show the wrapper script being typed; the topic is unclear	✓	✓
The first narration line introduces the site, not a reason to watch	✓	✓
`Could not resolve hostname` in the bastion video is a strong hook, buried	✓	✓
Open with the conclusion or a question (cold open)	✓	✓
Cut the typing and the waits between steps	✓	✓
Bigger font, highlight the key output lines	✓	✓
Make a vertical version	✓	✓
Insert the article's diagrams: permissions at `ls -l`, the before/after at the climax	✓	✓
End on the article's OG image	✓	✓
Post the bastion video first	✓	✓
Show a progress bar along the top	✓
Rewritten narration lines, as actual sentences		✓
The four `ssh-copy-id` INFO lines are noise		✓

The fixes themselves are standard short-video advice: open with the conclusion, cut silence, keep subtitles short, put the call to action at the end. Either model will give you that. This comparison didn't surface anything only K3 could find. I expected a model that reads video directly to catch something Claude couldn't see. It didn't.

Where they differed: timestamps

The biggest gap was in how precisely each one placed things in time.

K3 wrote things like "the first meaningful output appears at 3.7 seconds" and "the error at 00:03.2 in the bastion video gets buried," sub-second throughout. When I later had it review an improved version, it came back with intervals: "from 30.5 to 36.1 seconds the screen is frozen after the output appears."

Claude's timestamps were estimates from frames 2 seconds apart, and Claude said so at the end of its review: interval boundaries could be off by about ±1 second. You can sample more densely, but then there are more images to read. It's a trade-off.

"Where exactly is the screen frozen?" is hard to answer from a handful of stills. That's where reading the video directly paid off.

Effort and speed

	K3	Claude
My effort	Hand over the file	Decide how to sample frames (Claude redid it 4 times)
Time	363 s	185 s
Size	84,461 input / 10,863 output tokens, about $0.20 (part of the input billed at the cache rate)	111,313 tokens, 40 tool calls

K3 took less effort; Claude was faster. Each has its edge. Because K3 spends time reasoning, each review took 6-8 minutes. Enough time to make a coffee and come back.

The videos really did get better

I made the fixes. Following the reviews, I added six changes to handson-cast:

Open with a card: "Same key, but ssh -J gets rejected. Why?"
Cut the wrapper-script typing and the waits between steps
Type commands out character by character, with a larger font
Right after the rejection, show the article's diagram full-screen for 4.5 seconds
Once the output is complete, slowly zoom into the top-left of the terminal
End with the article's OG image and URL

Here's the final cut (v4, 41 s, with audio):

Click the image to open the mp4 (with Japanese narration).

Frames from the same timestamps, side by side:

The bastion video went from 46.8 to 41.0 seconds, about 6 seconds shorter. I had K3 grade the improved cut against a 10-item checklist for technical short videos. It covers things like a hook in the first 3 seconds, something changing on screen every 1-3 seconds, and a short call to action only at the end. It scored 7 pass, 2 borderline, 1 pass (estimated). The two borderline items were a frozen stretch after the output appears and an overlong subtitle. Both are fixed now.

I'd meant to have K3 research that checklist too, but that didn't work. The Kimi API has a built-in web search tool ($web_search). With K3, though, the second call, the one that returns the search results, failed every time with 400 tokenization failed. With kimi-k2.6 the first search went through, but after that it stopped searching and wrote text that only looked like search calls into its answer. The docs say this feature is scheduled for retirement on 2026-10-20 (Kimi API: Web Search). I gave up. Claude's web search did the research, and I passed the results to K3.

How I'll split the work

Based on this, here's how I plan to use them:

K3 when I need to know where in a video something happens. Frozen intervals, the moment the screen gets cut off, how long a caption stays up: questions whose answer is a span of time. I don't have to think about how to sample frames
Either one for deciding how to fix it. The improvement ideas didn't differ
Not K3 for anything involving audio. For narration content or sync, I prepare a transcript separately and compare against it
No confidential screens. The video goes to Moonshot AI's servers. These videos were going public anyway, so that was fine; internal screen recordings need different handling

It's a small comparison: two videos, one set of questions, nothing statistical. Still, it gave me a working answer. Reading video directly didn't change the judgment; it changed timestamp precision and effort. Next I want to plant deliberate defects in a video and count how many each model finds.

References

MoonshotAI/Kimi-K3 (GitHub): model card. 2.8T total parameters, MoE with 104B active per token, 1M context
Kimi API: Vision models: how video input works
Kimi API: Web Search: $web_search and its retirement date
bradautomates/claude-video: the /watch tool used for Claude's frame extraction
charmbracelet/vhs: terminal recording
SSH: five ways to reach a remote machine, compared on seven axes (Japanese): the /learn/ series these videos were made for

Shai-Hulud: 796 Packages, 1 Yes-Mash, and the Claude Code Auto-Approve Risk

Ken Imoto — Wed, 14 Oct 2026 13:00:01 +0000

On November 24, 2025, JFrog Security Research disclosed that a self-replicating npm worm had quietly backdoored 796 unique packages. The industry called the second wave Shai-Hulud 2.0, or "The Second Coming." It followed the September 16, 2025 original wave, which had already poisoned over 1,150 packages (JFrog, Datadog Security Labs).

Over 25,000 GitHub repositories had credentials leaked by the campaign (Sysdig). The 796 is the count of poisoned npm packages. The 25,000 is the count of victim repositories whose tokens and secrets the worm harvested; the attacker-side drop points were separate public repos (random 18-character names, description Sha1-Hulud: The Second Coming) created on victim accounts.

The infection vector was the same thing every developer does fifty times a day. Run a package install. Hit Enter.

Now add Claude Code, or any other coding agent, with "always allow" turned on for npm install. The fifty-times-a-day moves to 500-times-a-day. The attacker now needs one of those 500 to land on a compromised version.

That is the article.

What Shai-Hulud 2.0 actually did

The worm's loop was simple:

Infect. A victim ran npm install on a poisoned version of a popular package. The payload fired as a pre-install hook. (V1 used post-install; V2 moved it earlier, so it ran even on failed installs.)
Steal. The payload scanned the victim's machine for ~/.npmrc tokens, GitHub tokens from ~/.config/gh/, SSH keys, AWS and GCP credentials, and loose .env files. Credentials were exfiltrated by creating a public GitHub repository on the victim's account (description stamped Sha1-Hulud: The Second Coming) that acted as the drop point.
Self-replicate. Using the stolen npm token, the worm published poisoned versions of every package the victim had publish rights to. The victim became the next publisher.
Persist. In some cases it enrolled the victim's host as a self-hosted GitHub Actions runner named SHA1HULUD, so the attacker could execute code on the machine later (Netskope, Zscaler).

Three details matter for the agent-usage angle:

The payload ran on install, not on first use. There was no runtime signal.
The exfiltration channel was public GitHub, not an exotic webhook. Egress filters that block weird DNS wouldn't help.
The runtime was Bun, not Node. Signature-based scanners trained on Node payloads lagged (Endor Labs).

By the time npm removed the compromised versions, the token mesh was already seeded. The affected packages had roughly 132 million monthly downloads between them.

Why "always allow" is the exploited behavior

Claude Code's permission prompt is the exact friction Shai-Hulud needs you to remove.

Default behavior: Claude Code asks before running npm install <pkg>. You read the package name, you decide. One prompt per install. Annoying when you're installing twelve dependencies for a Next.js app, which is why /permissions exists and why most Claude Code users have Bash(npm install:*) on "always allow" by their third week. Mine was on by week two. I know this because I checked after reading the JFrog post.

The attacker's threat model relies on this. The payload does not need to defeat your judgment. It needs you to never be asked.

The equivalent risk on other agents:

MCP servers with auto-approve: true for all tools, so a server you installed last month can shell out and you would never see it.
Codex, Aider, or Cursor running under --yes or an auto-run shell wrapper.
CI: a bot account running npm install without lockfile integrity, auto-publishing on merge.

In every case the pattern is the same. A human decided once, early, then stopped looking. The worm does not need to be smarter than Claude Code. It needs the human in the loop to have left the loop.

A defensive checklist that isn't "write a CLAUDE.md"

CLAUDE.md helps against prompt-injection and model behavior. It does not stop a signed, published, poisoned package from running exec in a pre-install hook. By the time install scripts fire, the model is no longer in the loop.

The checklist below targets the install surface itself.

Per-session, before you say "always allow":

Run /permissions and read what is on always-allow right now. If Bash(npm:*) or Bash(*) is on the list, that is your exposure.
Scope allow rules to specific commands, not globs. Bash(npm ci) or Bash(npm install --package-lock-only) is survivable; Bash(npm install:*) is not.
For MCP: review auto-approve arrays per server. New MCP servers get a sandbox project first, not production.

At the install layer:

Pin everything. package-lock.json + npm ci + --ignore-scripts on CI where you can. Install scripts are the exact Shai-Hulud entrypoint.
Flip --ignore-scripts on globally in local dev (npm config set ignore-scripts true), then whitelist the handful of packages that legitimately need scripts.
Verify the registry you resolve from (npm config get registry) and the integrity hashes in your lockfile. The worm published to the real npm registry, so this does not save you from the attack itself, but it does catch tampering between your lockfile and your node_modules.

At the detection layer:

Socket, Snyk, or GitHub Dependabot wired into every repo. Socket in particular flags behavioral risks like install scripts and unusual network calls, and was fast on both Shai-Hulud waves.
Token hygiene. ~/.npmrc and ~/.config/gh/ are what the worm wants. Rotate npm tokens quarterly, use npm token create --read-only for CI, scope GitHub PATs down to the minimum.
Watch your account for the Nov 2025 exfiltration markers: public repos with randomly-generated 18-character names and description Sha1-Hulud: The Second Coming, workflow files at .github/workflows/discussion.yaml or formatter_*.yml, and self-hosted runners registered as SHA1HULUD. (The V1 wave in Sept 2025 used repos literally named Shai-Hulud and a workflow at .github/workflows/shai-hulud-workflow.yml.)

At the agent layer:

Do not leave always allow on across restarts for Bash, WebFetch, or anything that writes files outside the project. Per-session is survivable, forever is not.
Have the agent list the package name and the latest publish date before any install. A fresh version of a popular package deserves a human look, even when you are tired.

None of these are novel. All of them are standard. What Shai-Hulud 2.0 showed is that the "I'll just hit Enter" discount was always being subsidized by someone else's attention budget. The second wave took the subsidy away.

Trust, but verify

Reagan used it on arms control. It fits here. Claude Code is in the loop. Your other agents are in the loop. Both are useful. Neither replaces the one-second pause before you widen a trust boundary.

The industry has, so far, caught Shai-Hulud waves inside 48 hours of publish (CISA alert, Trellix). The vulnerability window is small, but it is the exact window your "always allow" paves over.

For a longer walk-through of Claude Code security patterns, including CLAUDE.md templates and data-retention policy per plan, I pulled these together in Practical Claude Code on Kindle.

Deepgram Nova-3 + Cartesia Sonic 3 + GPT-4o: Where 750ms of Voice AI TTFB Actually Hides

Ken Imoto — Tue, 13 Oct 2026 13:00:00 +0000

Voice-AI conversations die the moment the agent takes too long to say the first word. Not the full answer. The first byte of audio.

The industry calls this TTFB (time to first byte), or TTFA (time to first audio). I measure it from the moment the user stops speaking to the moment the first audio frame plays. Everything else, including total answer length and follow-up quality, happens after the user has already decided whether the agent feels alive.

Below is a ~750ms streaming budget on a Deepgram Nova-3 + GPT-4o + Cartesia Sonic 3 stack, broken into what each hop actually costs. And where almost every vendor-quoted number stops matching the number you will measure in production.

1,200ms to 750ms is the easy part

Batch pipelines wait for the full response before TTS starts:

[User stops speaking] → STT → LLM (full answer) → TTS (full) → [play]
                                                                 ^ >1,200ms

Streaming pipelines hand sentence 1 to TTS the moment the LLM emits it, while sentences 2 and 3 are still generating:

[User stops speaking] → STT → LLM (sentence 1) → TTS (sentence 1) → [play]
                               LLM (sentence 2) → TTS (sentence 2) → [continue]
                               LLM (sentence 3) → TTS (sentence 3) → [continue]

On the stack above, the streaming budget breaks down roughly like this:

Hop	Time	Notes
STT (Deepgram Nova-3 or Flux)	350ms	includes end-of-turn detection on Flux
LLM first sentence (GPT-4o)	150ms	first-token latency, best-case
TTS first chunk (Cartesia Sonic 3)	170ms	first audio frame
Network round-trips	80ms	user ↔ server + server ↔ vendor
Total (streaming TTFB)	~750ms

The 1,200ms to 750ms move pays for itself: no code change beyond "do not wait for the full response." The hard part is everything below 750ms.

STT: Nova-3 is 300ms, Flux adds end-of-turn for about 260ms

Deepgram's public numbers, cross-referenced with Softcery's 2026 comparison:

Deepgram Nova-3: streaming latency under 300ms, 36 languages. General ASR; does not detect end-of-turn.
Deepgram Flux: around 260ms end-of-turn, English only. Flux folds ASR together with turn detection into a single model and emits StartOfTurn, EagerEndOfTurn, EndOfTurn events (Deepgram launch post).

The combination matters. Nova-3 transcribes, but you still need a separate endpointing step to know when the user is done. Flux folds endpointing in, which cuts a 400-500ms round-trip, but costs you multi-language support.

For reference across other vendors: OpenAI gpt-4o-transcribe is around 320ms with 100+ languages. AssemblyAI Universal-3 Pro Streaming is under 600ms. ElevenLabs Scribe v2 Realtime is around 150ms with 90 languages.

Each of these is a vendor-quoted number. None of them is what you will measure from Tokyo over LTE to a US-East endpoint.

LLM: GPT-4o claims 200-400ms, measured 719-1,608ms

This is the hop where the vendor-vs-real gap gets ugly.

Deepgram's own blog post listing reasons they chose GPT-4o for voice agents cites a first-token latency of 200-400ms. The same post's measured table, on a MacBook M1 Pro over Wi-Fi, reports GPT-4o first-token times of 719ms to 1,608ms. That is 2x to 5x the vendor's number, on a laptop, on consumer Wi-Fi.

A separate Dev.to benchmark from CloudX tested 30+ stacks and found GPT-4.1-nano landing at 0.30-0.40 seconds on 2nd-turn-onward, where the KV cache is warm. First turn is always worse.

What the numbers teach: first-token latency is dominated by TTFT variance, which is dominated by model size, GPU type, quantization, input length, KV cache hit rate, inference backend load, and cold starts. A table of "model X equals N ms" is strictly less useful than one measurement from your own region to your own endpoint.

Running a small LLM locally removes the LLM-side network round-trip entirely. On a tuned setup, that is where sub-500ms TTFB actually lives.

TTS: Cartesia Sonic 3 says sub-90ms, measured median is 166-190ms

Cartesia's two models get swapped in half the voice-AI blog posts:

Cartesia Sonic 3: sub-90ms (vendor claim, measurement definition undisclosed), 40+ languages, emotion support.
Cartesia Sonic Turbo: 40ms (vendor claim, speed-specialized model), 9 languages.

The widely-circulated 40ms number is Sonic Turbo, not Sonic 3. Real Sonic 3 measurements:

Cartesia's own launch disclosure: median around 190ms.
Vapi's Humanness Index (50-run test, round-trip included): median 166ms.

Both numbers are roughly 2x the vendor-quoted sub-90ms. Neither 166 nor 190 is "wrong." They measure different things: vendor model-inference vs client-side round-trip.

ElevenLabs Flash v2.5 quotes around 75ms, with their own docs noting that 75ms refers to model inference only, and that end-to-end latency will vary with your location and endpoint type (ElevenLabs latency docs).

Research on streaming TTS reaches sub-50ms TTFA (arXiv 2508.04721). But the same paper's own end-to-end telecommunications voice agent, with the LLM in the loop, measured 678ms average TTFA. The TTS was fast. The stack still was not.

The 3-level optimization ladder

Treating these numbers as a budget, the ceiling moves in three steps.

Level 1, 500ms target: fast TTS + sentence-level streaming + streaming STT. This is the "do not wait for the full response" move above; gets you from 1,200ms+ to roughly 500-750ms.

Level 2, 300ms target: Level 1 plus Flux-style end-of-turn detection, speculative LLM invocation on partial ASR, a smaller LLM, and a TTS whose measured (not quoted) first byte is under 100ms. The emphasis is on measured.

Level 3, 200ms target: edge inference, a unified STT+LLM+TTS model, hardware tuning. At this point you stop composing vendor APIs and start building.

The 300ms figure is not arbitrary. It is the UX cliff below which a voice agent starts feeling like a human conversation partner, as opposed to a very fast IVR. Everything above 500ms feels like an echo.

Three rules before you trust any TTFB number in a blog post

Was the network included? ElevenLabs says the quiet part out loud (75ms is model inference only). Most vendor numbers do not.
Which model? Cartesia's 40ms is Sonic Turbo; the Sonic 3 you probably want is 2x that.
Who measured it? Vendor, third-party, or user-run on their own hardware? Each of the three routinely disagrees by a factor of 2.

Build your own measurement. Pick the stack above, or three alternatives on each row of the table, point at your actual region, record a 50-run distribution. The p95 is the number you should budget against. The vendor's median is marketing.

The full chapter this article compresses, including the TTS vendor-vs-measurement table and the 3-level ladder in more detail, is in The 300ms Threshold on Kindle. The book is built around voice-AI UX cliffs at 300, 500, and 800ms, and how to budget inside them.

Kernighan, Fowler, Torvalds: 7 Debugging Quotes Every Senior Still Repeats

Ken Imoto — Mon, 12 Oct 2026 13:00:00 +0000

Some engineering lines refuse to fade.

They survive because every senior ends up repeating them in code review, year after year, usually on a Thursday afternoon when a junior ships something that works and shouldn't.

I spent the last few months decoding a hundred of these for a short Kindle book. Below are seven, from three names that keep surfacing when the topic is debugging: Brian Kernighan, Martin Fowler, Linus Torvalds. Each quote has its original public source. Each one has a modern code-review move I use when I quote it.

1. "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"

Brian Kernighan & P. J. Plauger, The Elements of Programming Style, 2nd ed., ch. 2, 1978.

Half a century later, this is the single question I ask when a PR shows up with a one-line regex, a nested ternary, or a reduce that collapses six operations into one line. Not "is it correct." How will you ever debug it?

The AI-pair-programming era rediscovers this one every quarter. Give Claude Code or Copilot a prompt like "make this shorter" and it will happily compress four lines into one with a currying trick. The code works. Six months later, nobody (human or model) can rewrite the same function from scratch, because the compressed form hides the steps.

My review rule: if the author needs more than one sentence to explain what the line does, split it. I know, because I've been that author more than once.

2. "The most effective debugging tool is still careful thought, coupled with judiciously placed print statements."

Brian Kernighan, Unix for Beginners, 1979.

I used to be embarrassed by println debugging. Then I watched enough senior engineers reach for it before a debugger that I stopped apologizing.

The quote still holds, with one update. The tool today is console.log, fmt.Println, print(), or whatever your runtime offers, plus a tail of structured logs. Debuggers help when you can reproduce the bug and the state is static. Print statements help when the bug lives in a flow, in production, or at a boundary the debugger can't easily cross: gRPC, serverless cold start, worker queues, flaky async tests.

The actual skill the quote names is careful thought. The prints are only useful once you've guessed where to put them.

3. "Controlling complexity is the essence of computer programming."

Brian Kernighan & P. J. Plauger, Software Tools, p. 319, 1976.

A junior and a senior will both call code "complex." The junior usually means "I don't understand this yet." The senior usually means "nobody will ever understand this again, including the person who wrote it."

I keep this quote pinned when reviewing refactors. The question isn't "is this clever." It's "does the next on-call engineer, at 2 a.m., with no context, have a chance?" Most production bugs I've chased started as a cleverness-driven refactor that added coupling the diff didn't name.

Complexity accrues. Most PRs make it worse. The ones that make it better are rare and should be celebrated.

4. "Any fool can write code that a computer can understand. Good programmers write code that humans can understand."

Martin Fowler, Refactoring: Improving the Design of Existing Code, p. 15, Addison-Wesley, 1999.

This is the one line of Fowler's that engineers who've never read Refactoring can still recite. Fair, because it compresses the whole book: tests exist so humans can change code; names exist so humans can read it; small methods exist so a human can hold one in their head.

The habit I lifted from this: whenever I'm tempted to leave a comment explaining what a block does, I rename or split instead. If the code reads without the comment, the comment was a smell.

LLM code generators love long comments. Rename the function.

5. "If it hurts, do it more often."

Martin Fowler, martinfowler.com, FrequencyReducesDifficulty.

Fowler surfaces this as one of his favorite soundbites. The surface reading sounds like masochism. The real reading: if deploys are painful, deploy daily; if merges are painful, merge every hour; if integration tests are painful, run them on every push.

The pattern survives the current stack. Painful AI-code reviews? Review each PR as soon as it opens, not once a day in a batch. Painful dependency bumps? Automate them nightly instead of quarterly. Painful incident post-mortems? Hold them within 48 hours while people still remember.

Every time I've watched a team avoid a painful activity, the pain compounded. Fowler's rule flips the compounding.

6. "Talk is cheap. Show me the code."

Linus Torvalds, Linux Kernel Mailing List, 25 Aug 2000.

Linus wrote this in the middle of a kernel discussion where someone was proposing changes without an implementation. The adage has outlived its original thread.

Modern application: architecture debates on Slack where no participant has opened the file in question. Design reviews on a tool nobody has prototyped. "We should migrate to X" threads with no benchmark attached. The quote doesn't ban design. It bans debating before anyone has sampled the code. Open a branch. Commit a 30-line proof of concept. Attach the diff to the thread.

I keep a text expander snippet with this quote. I pull it out maybe once a month, and every time it ends the thread in under an hour.

7. "Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

Linus Torvalds, Git mailing list, 2006. (git mailing list via LWN, 2006-07-27)

The underrated debugging quote on this list. Most bugs that reach production aren't in the control flow. They're in the shape of the data: a nullable field someone assumed was present, a one-to-many relationship someone modeled one-to-one, a cache that lies about freshness, a queue that quietly drops ordering guarantees.

When a bug surfaces, the first question I ask is what the data actually looks like right now, and what the code assumed it would look like. More than half the time, the fix is a schema change or a type refinement, and the logic was never wrong. I lost a weekend last year "fixing" logic that was fine — the schema was lying about a nullable column.

Modern echo: in LLM pipelines, the hardest bugs don't live in the prompt. They live in the context layout the prompt silently depends on: what the retriever returned, in what order, with what metadata.

Why these seven, out of a hundred

The engineering world has more quotes than it has reviewed PRs. These seven survive translation across stacks: a 1978 line is still live in a 2026 Claude-Code-generated diff.

If this slice landed, the full set (100 quotes across 10 chapters — code aesthetics, bugs and failure, design, teams, innovation, learning, craftsmanship, open source, the AI era, and an engineer's life) is in 100 Engineering Quotes Decoded. One decoded quote at a time, so you can read a decoding on a coffee break and bring it into code review the same day.

5 Code Review Biases: Why Senior PRs Pass Faster Even When They Shouldn't

Ken Imoto — Mon, 05 Oct 2026 13:00:00 +0000

Meta researchers suspected the bystander effect was slowing their code reviews when a group — rather than one person — got asked to look at a change. They tested it, confirmed it, and shipped BystanderRecRnd, a recommender that assigns a single reviewer so the "someone else will take it" delay stops happening. The paper is worth reading on its own. It is also one of five biases that quietly decide which PRs pass fast and which ones get picked apart, independent of code quality.

The other four — halo, authority, framing, ownership — show up every day in reviews that look technical from the outside. This piece walks through each one with where it fires, and the mechanism that stops it. Not vigilance. A mechanism. Reminders to "be careful" survive about a week.

1. Halo effect: the senior PR passes faster

The halo effect is the oldest finding in applied social psychology, replicated so many times it stopped being news. In code review it shows up as this: the same diff, submitted by a senior and by a new hire, gets different scrutiny. The senior's code is "they probably have a reason." The new hire's is "they haven't learned yet."

The inverse also fires. An engineer who shipped a famous bug last year gets careful reviews on code that is objectively fine. The halo rots in both directions.

I have been on both sides of this. The senior PR with a design flaw that got waved through, and the junior-era PR with a reasonable choice that got picked apart line by line. Neither reaction was really about the code.

The amplifiers in a modern codebase are visible:

GitHub commit count, PR frequency, OSS contributions on the sidebar
Prior employer (ex-FAANG, maintainer of a known OSS project)
How carefully written the PR description is — people assume the writing style predicts the code

The mechanism

Hide the author from yourself for the first pass. Open the diff tab, not the PR summary. Form a technical read of the code before you look at who wrote it. The vigilance version — "I will try to be fair" — has been measured and does not work. The structural version — "I read the diff before I see the name" — does.

Pair it with a fixed checklist (security, performance, test coverage, readability). A checklist does not stop you from being charitable to the senior; it stops you from skipping items for the senior and inventing items for the new hire.

2. Authority bias: tech leads get LGTM on auto-pilot

Authority bias is subtle because it does not look like a bias. The tech lead's PR gets approved because, in most cases, the tech lead's PR is actually correct. The bias is the margin. It is the fraction of approvals where the reviewer thought "this looks off" and did not comment, because the reviewer assumed the lead had a reason.

Over a year that margin compounds. The architectural choices that quietly went unchallenged are the ones that survive into legacy debt.

How to use it as a reviewee

The flip side is a tool. When you want a proposal to land, borrow authority deliberately:

Cite the official docs or RFC for your design choice in the PR description
Reference industry patterns or a specific paper for an architectural change
Ask a trusted senior for a pre-review, then mention the feedback was already applied

The ceiling of this technique is low. Borrowed authority does not improve the code; it only lowers the friction of the review. Use it where you are confident the proposal is right and the friction is costing you time.

The mechanism against it

For reviewers: when the author is a senior, switch to question-form comments. "Is this intentional?" lowers the social cost of a reply from the author, so you get the clarification instead of eating the uncertainty. "This is wrong" from a junior to a staff engineer rarely gets sent. "Why this instead of X?" gets sent.

3. Bystander effect: reviews that never start

This is the Meta finding. When a review request fans out to a team instead of a person, the review starts later. Everyone assumes someone else will pick it up. Nobody does. The PR sits.

Meta's team at Facebook rebuilt reviewer recommendation specifically to kill this. One reviewer, named at request time, with the group as fallback. Review start time dropped.

The mechanism

Default to single-reviewer requests. Group requests are a fallback, not the primary mode.
When you must request a group, name a point person in the first comment. "@alice primary, @team as backup" removes the ambiguity even if the UI does not.
A review older than 24 hours gets a direct ping, not a reshare into the group. Resharing distributes the "someone else" button to more people.

4. Framing effect: same bug, different reaction

Dr. Michaela Greiler, who studied code review feedback at Microsoft and now consults on code review culture, notes that aggressive feedback tends to slow down fixes and drag down the quality of the author's next PR, even when the technical content of the comment is identical to a constructive version.

The same N+1 bug, two framings:

Negative frame: "This N+1 query will cause performance problems. Fix it."
Positive frame: "If you switch this to eager loading, the query count drops from N+1 to 2. On production traffic the difference should be noticeable."

The technical content is identical. The receiver's reaction is not. One sounds like a problem report. The other sounds like a tip. Michael Lynch, formerly of Google, has a longer set of guidelines for code review comments; three of them do most of the work:

Talk about the code, not the author. "This line can produce a NullPointerException" is reviewable. "You messed up here" is a feelings reply.
Attach a why. "Change this" without a reason triggers a defensive reply. "Change this because X returns null in the pagination edge case" triggers a fix.
Use questions where possible. "Could foo return null here?" opens the conversation. "This is a bug" closes it.

The mechanism

A review comment template with three fields — "what", "why", "suggestion" — forces the framing. The field with "suggestion" at the end makes the comment hard to leave hostile without actively choosing to.

5. Ownership bias: taking the review as criticism

The last one lives on the receiving side. If you wrote the code five minutes ago, your sense of ownership is at its peak, and every comment reads as a judgment of you, not of the diff. The defensive-reply reflex fires. You start arguing with suggestions that, read a day later, you would just accept.

I have the Slack history to prove it: suggestions I defended at 11pm, merged at 9am the next morning without changing a single argument against them. The code did not improve overnight. I did.

The mechanism

Submit the PR a few hours after you finish, or the next morning. Ownership fades with distance. Comments read differently after one night of sleep.
Reframe the artifact. The codebase is a team's shared product, not a portfolio of personal creations. The reframe sounds abstract and feels silly, until the first time you accept a suggestion that an hour ago you would have fought.
Reviewers: name one thing you liked in the PR before any comments. "The test naming is clean" is not politeness; it signals that the review is about the diff, not an attack on the author. Reviews that are 100% suggestions trip the ownership reflex.

Why mechanisms, not reminders

Across all five, the pattern is the same. Reminders to "be aware" of a bias do not survive the next sprint. Mechanisms do. Hiding the author on the first pass is a mechanism. A single-reviewer default is a mechanism. A comment template with a "suggestion" field is a mechanism. Submitting a PR the next morning is a mechanism.

Code review is a technical filter and a social situation at the same time, and the social half is the one that decides whether the technical half ever lands. If you want the full set — 20 biases that distort estimates, meetings, and hiring, each with a mechanism rather than a reminder — the book collects them in one place:

Cognitive Bias for Software Engineers

7 Steps to Ship a Knowledge Graph on Neo4j: Step 3 Ate Both My Weekends

Ken Imoto — Wed, 30 Sep 2026 13:00:00 +0000

Neo4j's official framework for building a knowledge graph is seven steps long. It reads like a well-behaved playbook: define the use case, pick your sources, design the ontology, model the data, ingest, query, operate. If you were reviewing it on a slide it would look completely reasonable.

Then you sit down on a Friday night to actually build one, and Step 3 quietly reaches out and eats the whole weekend.

Twice.

The first time I built a knowledge graph on Neo4j, the ontology-design step ate two full weekends before I could even start writing the first Cypher CREATE statement. Steps 1, 2, 4, 5, 6, and 7 combined took less time than Step 3 by itself. That is not because the other six steps are trivial. It is because Step 3 is where every unresolved assumption from Steps 1 and 2 comes home to be reconciled, and you cannot fake your way past it.

Below is the 7-step arc as Neo4j teaches it, with what actually happens between the diagrams.

The 7 steps, at a glance

Step	What Neo4j calls it	What actually happens
1	Define the use case	You pick one query you want fast
2	Identify data sources	You classify structured / semi-structured / unstructured
3	Design the ontology	You loop back to Step 1 three times
4	Model the data	You write Cypher and remember indexes
5	Ingest and transform	LOAD CSV, APOC, LLM extraction
6	Build queries and APIs	Cypher becomes your business logic
7	Operate and extend	Freshness, quality monitoring, schema evolution

Steps 1 and 2 look easy on paper. Step 3 is the interesting one. Steps 4-7 are where the real technology sits.

Step 1: define the use case (or watch the whole project rot)

The bad version of a use case:

"We want to put all our internal data into a knowledge graph."

The good version:

"A new engineer needs to know, in 30 seconds, which services depend on /api/users before they change it."

What separates the two versions is not scope but specificity. A specific query pins down three things that would otherwise stay loose for months: which entities you need as nodes, which relationships you need as edges, and which query pattern has to be fast.

The industry has a graveyard full of KG projects that started with "let us model everything." Neo4j's own guidance is blunt about this: if the use case is not concrete, the graph is decoration.

Step 2: identify data sources (structured, semi-structured, unstructured)

The classification is the whole step:

Structured — CSVs, relational tables, customer masters, product catalogs.
Semi-structured — JSON API responses, config files, OpenAPI specs.
Unstructured — PDFs, meeting notes, source code, chat logs.

The reason this classification matters is that it decides how you get into Step 5. Structured data walks into Neo4j through LOAD CSV. Semi-structured comes in via APOC. Unstructured needs an LLM extractor between the source and the graph.

In September 2026 the LLM-extractor side of this has become a solved product problem: Neo4j's own LLM Knowledge Graph Builder requires Neo4j 5.23+ and now supports OpenAI, Gemini, Claude, Llama 3, Amazon Nova, Diffbot, and Qwen as the extraction model. The neo4j-graphrag-python SimpleKGPipeline turns on structured-output extraction when the LLM supports it, which cuts the "the model returned invalid JSON" retries you would otherwise write yourself. If you are starting today, you build the extractor last and use these tools.

Step 3: ontology design (the one that ate the weekends)

This is where the trouble happens. On paper, an ontology looks like a tiny document:

Node labels:
  - Service (name, version, team)
  - API (path, method, status)
  - Developer (name, email, team)
  - Repository (name, url, language)

Edge types:
  - EXPOSES:    Service -> API
  - CALLS:      API -> API
  - MAINTAINS:  Developer -> Service
  - HOSTED_IN:  Service -> Repository

That is maybe 15 lines. How did that eat two weekends?

Because writing those 15 lines forces you to answer questions Step 1 quietly deferred. Is a deployed instance of a service a Service or a Deployment? Is a Repository really one node, or one per branch? Does a MAINTAINS edge point from the developer to the service or the other way around, and does the direction change what queries become expensive?

The moment you start writing Cypher against a draft ontology, you find edges that need properties, node labels that need to split in two, and relationships whose direction you had wrong. You go back to Step 1, reread the use case, and change Step 3. Then you go back to Step 3 and change it again. The rough pattern I ended up with: Step 3 makes you loop back to Step 1 about three times before it settles.

The lesson I wish I had absorbed earlier:

Do not design the ontology in isolation. Sketch it in the same file as three sample Cypher queries you actually plan to run. If any of the three cannot be written cleanly, the ontology is wrong, not the query.
Start small. Two node labels and one edge type is a legitimate first iteration. Expanding is easy; unpicking is a weekend.
Read edges as sentences. "Service EXPOSES API." "Developer MAINTAINS Service." If you cannot say the edge out loud without it sounding weird, the direction is off.

I would also, in retrospect, have used Neo4j's Graph Consolidation feature earlier. It rolls a sprawling schema into fewer, more meaningful node labels and relationship types automatically. On a first attempt this is exactly the kind of judgement I did not yet have.

Step 4: data modelling in Cypher

Once the ontology settles, Step 4 is mechanical:

CREATE (:Service {name: "UserAPI", version: "2.1", team: "Platform"});
CREATE (:Service {name: "AuthService", version: "1.5", team: "Security"});
CREATE (:API {path: "/api/users", method: "GET", status: "active"});

MATCH (s:Service {name: "UserAPI"}),
      (a:API     {path: "/api/users"})
CREATE (s)-[:EXPOSES {since: "2024-01-15"}]->(a);

CREATE INDEX FOR (s:Service) ON (s.name);
CREATE INDEX FOR (a:API)     ON (a.path);

The single most common mistake at this step: forgetting the indexes. A KG with millions of nodes and no index on the query's start-node property will take seconds per query, and you will spend a week thinking your ontology is wrong when the real problem is a CREATE INDEX line you never wrote.

Step 5: ingestion and transformation

For structured data, LOAD CSV WITH HEADERS is the whole story:

LOAD CSV WITH HEADERS FROM 'file:///services.csv' AS row
CREATE (:Service {
  name:    row.name,
  version: row.version,
  team:    row.team
});

For unstructured data, the modern move is to hand the LLM a schema-constrained extraction prompt rather than a free-form one. SimpleKGPipeline in neo4j-graphrag-python does this for you when the underlying model supports structured output; you get JSON that conforms to your ontology rather than JSON that hopes to. The retries you save at this step add up.

Step 6: queries and APIs

This is where the graph starts paying rent. The three query shapes I ended up using most:

Blast radius — what breaks if I change this API?

MATCH (target:API {path: "/api/users"})<-[:CALLS]-(caller:API)<-[:EXPOSES]-(s:Service)
RETURN s.name AS affected_service,
       caller.path AS via_api;

Shortest path between two services.

MATCH path = shortestPath(
  (a:Service {name: "UserAPI"})-[*]-(b:Service {name: "PaymentService"})
)
RETURN path;

Community detection — which services cluster together?

CALL gds.louvain.stream('service-graph')
YIELD nodeId, communityId
RETURN gds.util.asNode(nodeId).name AS service, communityId
ORDER BY communityId;

Cypher is where a knowledge graph stops being a nice diagram and starts being business logic. Those three shapes cover most of what you actually reach for once the graph is live.

Step 7: operate and extend

The step nobody wants to write about, because it does not screenshot well.

Freshness. The source data changes; your graph does not, unless you have a sync pipeline.
Quality. Orphan nodes and duplicate entities creep in. Schedule a job to find and merge them.
Schema evolution. New use cases mean new node labels and new edges. Version your ontology.
Access control. Not every team should see every subgraph.

Neo4j AuraDB shoulders most of the infrastructure side of this, which is why a lot of small teams end up on Aura even if they started on a self-hosted instance. The trade is real: less flexibility, less ops.

What I would do differently

If I were starting over today with the same use case:

Write the three killer queries first, before the ontology. The queries constrain the ontology far better than the ontology constrains the queries.
Cap the first ontology at two node types and one edge type. Expand only when a query forces the expansion.
Use SimpleKGPipeline (or the LLM Knowledge Graph Builder if you want a UI) instead of hand-writing extraction prompts. The structured-output layer alone saves days.
Do not skip the indexes. Ever.
Budget for Step 3 to take longer than Steps 1, 2, 4, 5 combined. If it does not, either your use case is unusually narrow or you are about to find out you missed something.

The 7-step arc is honest. It is the shape of the effort inside each step that the diagram does not capture. Once you know Step 3 is going to be the weekend eater, you can plan around it — you sketch queries alongside the ontology, you keep the first pass small, and you accept that "the graph is designed" is a claim you cannot make until you have written Cypher against it.

If you want the longer version, my Zenn book Knowledge Graph Practical Guide walks the full arc, and Chapter 4 is exactly this 7-step build — including the ontology-loop story that turned my weekends into a lesson.

References

Neo4j — LLM Knowledge Graph Builder (Neo4j Labs)
Neo4j — LLM knowledge graph builder — First release of 2025
Neo4j — User Guide: Knowledge Graph Builder (neo4j-graphrag-python)
Neo4j Labs — llm-graph-builder GitHub releases

Local LLM vs Claude Code: 96% of My Requests Failed Locally. Half the Steps Didn't.

Ken Imoto — Tue, 29 Sep 2026 19:53:48 +0000

The pitch for running Claude Code on a local model goes like this: point it at Ollama, stop paying per token, keep your code on your machine. I have an RTX 4070 with qwen3.5:4b on it, so I wanted that to be true.

Before swapping anything, I counted. I took 100 requests I had actually sent to Claude Code and asked, for each one, whether a local model could have done it end to end.

96 could not. So much for cancelling the subscription.

Then I counted a different way, and the answer flipped. This post is about the second count and the small config file that came out of it.

Counting whole requests: 96 of 100 go to the frontier

A request to Claude Code is usually one line. "Fix the failing test in the billing module." "Why is this deploy slow?" The request itself contains very little context. The work happens after that, when the agent reads files, runs commands, fetches docs, and decides what to try next.

So when you judge the request as a whole, you are judging the hardest part of it: the planning. That part needs the frontier model almost every time. In my set, 96 of the 100 had to go there.

If you route at the request level, a local model gets almost nothing. Swapping the whole agent onto a local model asks it to do exactly that planning.

Counting single steps: 97 of 200 run fine locally

The agent's work is a chain of steps: tool calls, fetches, summaries, commit messages. I broke Claude Code sessions into 200 of those steps and asked the same question for each one.

97 of 200 were fine on a local model.

The clearest case was WebFetch: fetch a page, pull out the part that answers a question. That is extraction, not planning. I tried it on 20 real fetches, and it was not perfect: some pages never loaded, and some answers came back partly wrong. That is why the example config sends a failed fetch to the frontier (on_fetch_error: frontier). But when the page arrived, the local model usually got the extraction right.

Claude Code already does a version of this. Its own WebFetch tool description says the fetched page is processed with a small, fast model. That makes the tool itself an example of choosing a model per step.

One rule beat a 9B router

My first router asked a local model for a probability: "should this step go local?" It worked, sort of. Then I wrote a one-line rule, "WebFetch goes local", and the rule beat the 9B model's probability.

That changed the design. Rules decide what rules can decide. When enabled, the probability judge only sees what is left. Here is the rules block from the companion repo's example config:

rules:                      # step rules, top to bottom, first match wins
  - match: {tool: WebFetch}
    route: local
  - match: {tool: [Agent, Task]}
    route: frontier
  - match: {kind: commit_message}
    route: frontier         # frontier until a probability judge is added

probability: null           # off until measured with the same judge model
default_route: frontier

The probability judge is off by default. The routing function is short:

def route(self, step: dict) -> Decision:
    for r in self.rules:
        if r.matches(step):
            return Decision(r.route, "rule")
    if self.probability and self.local_threshold is not None:
        p = self.probability(step)
        return Decision("local" if p >= self.local_threshold else "frontier", "probability", p)
    return Decision(self.default, "default")

In my config, sub-agents (Agent, Task) are pinned to the frontier.

Getting a probability out of Ollama

For the steps rules can't settle, you want a number, not a paragraph. The trick is to make the model answer with one letter (Y or N) and read that letter's probability from logprobs.

This is the request body I send to Ollama's native /api/chat:

{
  "model": "qwen3.5:4b",
  "stream": false,
  "think": false,
  "logprobs": true,
  "top_logprobs": 20,
  "options": {"temperature": 0, "num_predict": 1},
  "messages": [{"role": "user", "content": "... Answer with a single letter, Y or N."}]
}

Three things fail silently here, with no error:

The OpenAI-compatible endpoint drops the probabilities. Use the native API. The repo's scripts expect Ollama 0.12.11 or later.
Thinking models don't answer in the first token. The first token is the start of their reasoning, so the Y/N you are looking for isn't there. think: false and num_predict: 1 pin it to the answer.
A candidate outside the top 20 looks like probability 0. Also, " Y" and "Y" are different tokens.

The fix for the third one is to add up the probability that landed on your candidate letters and refuse to trust the judgment when that total is small:

raw = {lab: 0.0 for lab in labels}
for t in response["logprobs"][0]["top_logprobs"]:
    tok = t["token"].strip()  # " Y" and "Y" are the same answer
    if tok in raw:
        raw[tok] += math.exp(t["logprob"])
mass = sum(raw.values())      # below 0.5: the answer wasn't where we looked

0.5 is a line nobody chose

I used the same kind of judge as a gate in front of Claude Code: "does this request contain a secret?" It read a note with a production database password in it and said it was fine, with 69% confidence. The password was there with 100% confidence.

The ranking was fine. Secrets scored higher than non-secrets. The scale was off: real secrets were landing well below 0.5. Cut at 0.5, the gate missed 20 secrets. When I drew the line from my own labelled examples instead, it missed 2.

That is why the gate threshold in the example config looks strange:

judge:
  model: qwen3.5:4b
  threshold: 0.0071    # example only. lowest P among your confidential examples x 0.3
  at_or_above: block
on_timeout: block

0.0071 is only an example from my setup, so don't copy it. The repo has a small CLI (steprouter-threshold) that takes your labelled judgments and prints a line, AUROC, and 5-fold cross-validated misses and false alarms.

A regex pass (gitleaks plus personal-data patterns) runs before the model. It catches secrets that have a recognizable shape. The model is there for the ones that don't.

The free local worker cost the most

One more result surprised me. I set up a strong model to plan and a free local model to do the work. That setup was the most expensive one I tested. Free, the way a puppy is free.

The local model's tokens were free. The orchestrator re-reading everything the worker sent back was not. What comes back from a routed step is part of the routing, so the example config pins the return value to a small typed shape:

return:
  fields: [green, mypy, ruff, pytest_failed, files, summary]
  summary_max_lines: 2
  full_log: file

Pass/fail counts, the files touched, and one or two lines of what was done. The full log is written to a file instead of the reply.

Try it on your own logs

My numbers come from my work. To get yours:

Take a day of Claude Code sessions and list the steps, not the requests.
Mark which step types always go one way. Those become rules.
Leave the probability judge off until you have measured it on your own examples.
Before you add a local worker, decide what it is allowed to send back.

The companion code, including the gate, the logprobs judge, the threshold CLI, the router, and a made-up evaluation set, is on GitHub as local-step-router (MIT). Clone it, run pip install -e ., and start by editing examples/routing.yaml. The README has an English summary at the bottom.

The full measurement (550 of my own requests and steps on one RTX 4070, how big the judge model needs to be from 0.8B to 35B, why none of my six real secrets matched the evaluation set, and the final blueprint) is written up in Local LLM or Claude Code?. Chapters 2 and 3 cover the step count and the rules-first router; chapter 8 is the threshold worksheet.

200ms Humans vs 700ms Voice Agents: What ACL IWSDS 2025 Says About the Turn-Taking Gap

Ken Imoto — Wed, 23 Sep 2026 13:00:00 +0000

The complaint that keeps showing up in every voice-agent beta test I've watched is not about the wrong answer. It's about the timing of the right answer.

"The AI cut me off." "It answered while I was still thinking." "I said 'um' and it started talking." Latency looks like a stopwatch problem, but users experience it as a manners problem, which is a much harder thing to fix.

The 200ms vs 700ms gap the field is trying to close

The ACL IWSDS 2025 survey on turn-taking in spoken dialogue puts a concrete number on the mismatch. In natural human conversation, a listener starts their reply roughly 200ms after the speaker's turn ends. Current dialogue agents take 700 to 1000ms to do the same thing.

Three to five times slower. If human dialogue is a tennis rally, current voice AI is playing chess by post.

Closing that gap is not one problem, it's two. And they pull in opposite directions.

Too slow to reply: awkward silence, dropped rapport, users start repeating themselves.
Too fast to reply: the agent talks over the user, interrupts mid-word, kills trust.

Both fail modes come from the same root cause: silence is not the same as end-of-turn.

Why plain VAD is not enough

Most current systems still decide the user is done talking with a Voice Activity Detector. VAD is cheap and fast, and it answers exactly one question: is there voice in this audio frame, yes or no.

That question is not the one the agent actually needs answered.

The 0.3–0.5s pause a speaker takes mid-sentence looks identical to end-of-turn.
The silence after "uh" or "let me think" looks identical to end-of-turn.
A cough or a sigh looks like voice.

You can tune the silence threshold, but you cannot tune your way out of the fact that VAD does not know what a sentence is. VoiceInfra's production data lands on 300–500ms of trailing silence as the least-bad setting: shorter and the agent chops off natural pauses, longer and the delay becomes obvious. The comfortable range depends on the use case. Call-center flows tolerate 400–500ms because callers pause to think; command interfaces want 200–300ms because utterances are short; long-form narration wants 500–600ms because the pauses inside the story are longer.

The threshold is a compromise, not a fix. The real fix has to know what the words mean.

Semantic endpointing: Deepgram Flux and the "one model does both" bet

The interesting move in 2026 is folding transcription and turn detection into the same model. Deepgram's Flux, launched as their first ASR built for voice agents rather than for general transcription, does exactly that. The model outputs turn boundaries directly from audio, using the same joint architecture it uses to output text. Because the same weights see both the acoustic signal and the emerging transcript, the model has a shot at answering the harder question: "is this utterance semantically complete?"

Flux exposes three knobs that make the tradeoff explicit rather than hidden inside a silence threshold:

eot_threshold — how confident the model has to be before it commits to end-of-turn.
eager_eot_threshold — a lower bar for tentative end-of-turn, useful for starting inference speculatively.
eot_timeout_ms — the maximum silence duration before the model forces end-of-turn even if the confidence never crosses the threshold.

You can also switch modes per-turn with a Configure message, so a barge-friendly interaction ("agent, wait") and a monologue turn ("read me the terms") do not have to share a threshold.

The pattern is a hint at where the field is going. Turn-taking will stop being a pipeline stage after ASR and start being a joint output of the ASR itself.

Krisp's 6M-parameter turn model: same idea, edge-shaped

Not everyone can afford a cloud round-trip on every silence check. Krisp ships two small audio-only models trained to handle the acoustic cues a naive VAD collapses — a 9M-parameter turn prediction model (~30MB) that scores end-of-turn from prosody and pausing patterns, and a 6M-parameter interruption model (~24MB) that separates real barge-ins from backchannels like "yeah" and "mhm". Together they cover the same failure modes VAD misclassifies:

intentional speech vs a thinking pause
filler words (um, uh, well) mid-sentence
backchannels vs interruptions

At that size it runs on-device in real time, which matters for privacy-sensitive deployments and for wearables where the round-trip to the cloud is itself the biggest source of latency.

The Krisp and Flux paths look opposite. One shrinks the model, the other gives the ASR the extra job. But they are attacking the same VAD failure mode from opposite sides.

Graceful abort: what to do when you're wrong

No detector, semantic or acoustic, is going to be right every time. The interesting question is what your pipeline does when it decides "the user is done" and turns out to be wrong.

Twilio's graceful abort pattern is the cleanest version I've seen written up. When STT signals end-of-turn early, the LLM starts generating a response. If fresh audio arrives before the response reaches the speaker, the pipeline kills the generation in flight and swallows the partial output. The window for this is the few hundred milliseconds it takes STT → LLM → TTS to produce audible speech. If you can revoke the guess inside that window, the user never hears it.

The economics matter here: you are paying for LLM tokens on turns that get thrown away. In exchange, you get to be aggressive on end-of-turn detection without punishing the user when you're aggressive-and-wrong. For most conversational products that tradeoff is worth it.

Barge-in: hearing the user through your own voice

The other half of turn-taking is the reverse case: the user interrupts while the agent is still speaking. Detecting that seems trivial until you realize the AI's own audio is leaking into the microphone, and any naive VAD will happily flag that leak as a barge-in and abort the agent's own utterance.

Sparkco's write-up on duplex barge-in handling is the clearest description of the fix I've read. Three moving parts:

Full-duplex audio — keep the mic hot the entire time the agent is speaking, never gate it on TTS output.
Echo cancellation — subtract the speaker output (as a reference signal) from the mic input, so what remains is the user's voice minus the agent's.
Nuisance rejection — filter environmental noise and short transients that a bare VAD would misclassify as speech onsets.

[agent is speaking]
  speaker out ──► reference signal ─┐
                                    ▼
  mic in ─────► echo canceller ────► user voice only ──► VAD / turn model

Without echo cancellation you get the failure mode users describe as "the AI got startled by its own voice and stopped talking." The dog scared of its own reflection.

The bot-feel paradox and the 200-300ms delay trick

Now the counterintuitive move. Once you've closed the acoustic and semantic gaps and the agent can reply in 300ms, it turns out that replying in 300ms feels worse. The response arrives before the user has finished processing their own sentence, and it registers as robotic rather than sharp.

The fix is to inject 200–300ms of intentional delay before the agent starts speaking, while continuing to run the LLM in the background. You get the "thinking for a moment" cue humans read as attention, without paying real latency for it. The tokens are already streaming; you're just holding the TTS start.

It's a UX trick, not a technical one. But it's the piece of the puzzle that gets forgotten when a team spends a quarter shaving milliseconds off ASR and then wonders why user ratings didn't move.

What actually closes the 500ms gap

There is no single component that takes you from 700ms to 200ms. You get there by stacking:

Layer	What it buys you
Semantic endpointing (Flux-class)	Stops chopping mid-sentence, stops waiting for silence that already means end-of-turn.
Small on-device turn + interruption models (Krisp-class)	Score end-of-turn and separate backchannels from real barge-ins without a round-trip.
Graceful abort	Lets you be aggressive on early end-of-turn without punishing the user when wrong.
Duplex barge-in with AEC	Lets you leave the mic hot without the agent interrupting itself.
200–300ms intentional delay	Buys back the "thinking" cue that pure speed removes.

The move that unlocks the stack is admitting that end-of-turn is a language problem, not a silence problem. Once you accept that, VAD-only pipelines look the way pre-BERT NLP pipelines look now: perfectly reasonable for their era, obviously incomplete in retrospect.

The gap to 200ms is not going to close by tuning thresholds. It closes when the model that hears the audio is the same model that understands the sentence.

The full 300ms-UX playbook (latency budgets by pipeline stage, when to pick Whisper vs Deepgram vs Piper, and how to design the wait states so users forgive the last 200ms you can't remove) is written up in Voice AI 300ms UX. Chapter 9 covers turn-taking end-to-end; chapters 4 and 5 cover the latency anatomy that decides whether closing the gap is even possible for your stack.

References

ACL IWSDS 2025. "A Survey of Recent Advances on Turn-taking Modeling in Spoken Dialogue."
Krisp AI. "Audio-only, 6M weights Turn-Taking model for Voice AI Agents." 2025.
Twilio. "Core Latency in AI Voice Agents." 2025.
Deepgram. "Flux: End-of-Turn Detection Parameters." 2026.
Sparkco. "Optimizing Voice Agent Barge-in Detection." 2025.
VoiceInfra. "Voice AI Prompt Engineering: Complete Technical Guide." 2025.

Claude Code + Playwright MCP + GitHub Actions: 3 Test Layers That Actually Kill Regressions

Ken Imoto — Mon, 21 Sep 2026 13:00:01 +0000

The first time I asked Claude Code to write tests for a feature it had just implemented, the tests passed. All of them. On the first try. I felt great about it for about ten minutes, until I realized the tests were describing the code, not the requirements. The code had a bug I had specified against, and the tests happily locked the bug in.

That is context pollution. It is the fundamental problem with letting one model both implement and verify. The New Stack put it best: letting the model that wrote the code write the tests is asking a student to grade their own homework.

The fix is architectural, not attitudinal. Split the loop across three isolated contexts, each running at a different point in the development lifecycle. Here is what those three layers look like when you build them out.

Layer 1: TDD by subagent (dev time)

Red-Green-Refactor works with Claude Code, but not the way most teams first try it. The naive setup is one Claude Code session doing all three phases. That is the setup that produces homework-grading.

The working setup uses three subagents, each blind to the others' work.

Skills (orchestrator)
  ├── tdd-test-writer   (RED)    — sees feature description, NOT implementation
  ├── tdd-implementer   (GREEN)  — sees test file, NOT the writer's reasoning
  └── tdd-refactorer    (REFACTOR) — sees passing code, must keep tests green

The critical bit is that each subagent runs in its own context. The test writer cannot peek at how the implementer plans to solve the problem. The implementer cannot see the notes the test writer wrote about "edge cases we should probably cover."

A minimal skill definition for the writer:

# .claude/agents/tdd-test-writer.md

You are a test writer. Your ONLY job is to write failing tests.

Rules:
- You CANNOT see or reference any implementation files
- Write tests based ONLY on the feature description
- Tests must cover happy path + edge cases (nulls, empty, negative, overflow)
- Run tests and confirm they FAIL before finishing

And the implementer, which is deliberately narrow:

# .claude/agents/tdd-implementer.md

You are an implementer. Your ONLY job is to make tests pass.

Rules:
- Write the MINIMUM code to make tests pass
- Do NOT refactor or optimize (that's the refactorer's job)
- Do NOT modify test files
- Run tests and confirm they PASS before finishing

Why this works: the "design intent" that a normal Claude Code session leaks between phases now cannot leak. The test author writes what the feature should do; the implementer writes what makes those tests pass; the refactorer cleans up without breaking anything. It is exactly the discipline TDD was invented to enforce, made mechanical by the fact that context boundaries are hard walls now.

The other nice property: you can CLAUDE.md this policy at the repo level and forget about it.

## Testing policy
- All new features go through the TDD subagent chain
- No test code and implementation code in the same Claude session
- Skip TDD only for one-line typo fixes and docs

Layer 2: Quinn, the Playwright MCP QA agent (PR time)

The second layer runs at pull-request time and does the thing no unit test can do: click around the UI like a person who has never seen the code.

Alexop's Quinn pattern is the reference here. Quinn is an AI QA engineer defined by a prompt: a 12-year-veteran QA persona, instructed to test only through the UI, never look at source, always exercise the mobile viewport (375x667), and file findings with reproduction steps.

The mechanism that makes Quinn possible in 2026 is Playwright MCP. The server ships dozens of browser-automation tools exposed over MCP, and the important detail is that it hands the agent an accessibility tree as structured JSON rather than a screenshot. No vision model needed. Claude Code reads the tree, decides what to click, tells Playwright to click it, reads the resulting tree, and so on. Deterministic, cheap, and stable across page redesigns.

A working .github/workflows/ai-qa.yml:

name: AI QA (Quinn)
on:
  pull_request:
    types: [labeled]

jobs:
  qa:
    if: github.event.label.name == 'qa-review'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install && pnpm build
      - name: Start dev server
        run: pnpm dev &

      - uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          claude_args: |
            --mcp-config '{
              "mcpServers": {
                "playwright": {
                  "command": "npx",
                  "args": ["@playwright/mcp@latest", "--headless", "--no-sandbox"]
                }
              }
            }'
          prompt: |
            You are Quinn, a senior QA engineer with 12 years of experience.
            Test the application at http://localhost:3000.
            DO NOT look at source code. Test only through the UI.
            Test on desktop (1280x720) and mobile (375x667).
            Try edge cases: empty inputs, huge inputs, negatives, unicode names.
            Report each bug with a screenshot and a reproduction sequence.

Two flags in that config carry more weight than they look: --headless --no-sandbox on the MCP server side makes it run under GitHub Actions' container without display. Claude Code itself runs non-interactively under claude-code-action@v1, so the action handles the "exit when done" part for you.

Quinn is not a replacement for scripted E2E tests. Scripted E2Es catch known-bad regressions. Quinn catches the class of bug where "nobody thought to write a test for that particular sequence." Two different tools, two different failure modes.

The trigger being types: [labeled] matters. Quinn burns real tokens, and running her on every push to every PR gets expensive. Making her opt-in via a qa-review label means the team pays for her when the PR is close to ready, not while it is still churning.

Layer 3: claude-code-action on the CI gate (merge time)

The third layer is the boring one, which is why it works. Once a PR passes Quinn, one more Claude Code invocation runs on the CI gate, looking at the diff against the existing test suite.

# .github/workflows/claude-test.yml
name: Claude Code Test Gate
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - run: pnpm install

      - uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          prompt: |
            Analyze the changes in this PR. Then:
            1. Run the existing test suite: pnpm test
            2. Identify code paths in this diff not covered by any test
            3. Propose specific test cases for the gaps (do not write them yet)
            4. Post the analysis as a PR comment

This is the "did we miss anything" pass. It does not write tests (that would push us back into homework-grading), it flags gaps. A human reads the comment, decides which gaps matter, and either accepts them or opens follow-up tickets.

Anthropic ships claude-code-action@v1 as the official integration, and the setup is one CLI call away — /install-github-app from Claude Code walks you through installation, permissions, and mentioning @claude on issues or PRs to get responses.

Between the three layers, the diff of who runs which tests when looks like this:

Layer	When	Who tests	What they use
1. TDD subagents	Dev time (per commit)	Claude Code, subagent per phase	Feature description + failing test
2. Quinn (Playwright MCP)	PR time (label-triggered)	Claude Code as QA persona	Browser via MCP, no source access
3. claude-code-action	Merge time (every push)	Claude Code as reviewer	Diff + existing suite + coverage report

Notice what each layer has that the others don't. Layer 1 has the requirements but not the implementation. Layer 2 has the running app but not the source. Layer 3 has the diff but is forbidden from writing tests. Each layer's blindness is what makes it useful.

Why selector reliability improves under this shape

Small side-benefit worth mentioning. When Quinn writes down what she clicked (for the reproduction steps she files with bugs), she writes it in Playwright MCP's accessibility-tree vocabulary. That means:

// The kind of selector Quinn produces
await page.getByRole('button', { name: 'Log in' }).click();

// Not the kind you get from asking Claude to "look at the DOM"
await page.click('.btn-primary.mt-4.px-6');

The role-plus-text form survives redesigns, works with screen readers, and doesn't break the first time someone renames a Tailwind class. When you promote Quinn's reproduction steps into permanent E2E tests (which you should, whenever the bug is worth guarding against), you get selector stability for free.

What each layer is doing that the others can't

The reason context pollution is a whole category of AI-testing bug is that most teams try to solve testing with more AI in one place. More prompts, better instructions, longer CLAUDE.md rules about "please check your own work." None of that helps. The problem is information leakage between phases, and stronger prompts inside one leaky pipeline still leak.

Three isolated contexts, three different data diets:

The test writer eats requirements, produces tests.
Quinn eats the running app, produces bug reports.
The CI reviewer eats the diff, produces gap analysis.

Roll any two of those together and you get a subtle regression somewhere. Keep them separate and each does its narrow job well.

The Anthropic tooling to make this work — subagents, Playwright MCP, claude-code-action — all landed in 2025-2026. The pattern was possible before, in the way most patterns are "possible" before the tooling; now it is a config-file exercise, not a research project.

If your team already has one Claude Code session doing all three jobs, split them next sprint. The tests will start finding things the single-session setup silently endorsed.

The full Claude Code testing playbook (subagent memory design, when to use skills vs commands vs subagents, how to structure a repo so the three layers stay independent, and what to do when Quinn files a false positive) is written up in Claude Code Mastery. Chapter 10 covers test automation end-to-end, including the TDD-subagent split this article is a distillation of.

References

The New Stack. "Claude Code and the Art of Test-Driven Development." 2025.
alexop.dev. "Building an AI QA Engineer with Claude Code and Playwright MCP." 2025.
alexop.dev. "Forcing Claude Code to TDD: An Agentic Red-Green-Refactor Loop." 2025.
Anthropic. "Claude Code GitHub Actions." Official docs, 2026.
Microsoft Playwright team. "Playwright MCP Server." github.com/microsoft/playwright-mcp, 2026.

MCP Resources vs Tools vs Prompts: 3 Layers That Cut My Agent's Tokens From 114K to 27K

Ken Imoto — Tue, 15 Sep 2026 13:00:01 +0000

Most MCP posts on this site stop at the Tools layer. That is the layer everyone builds first: get_weather(), send_email(), query_database(). It is also the layer that will eat your token budget if you leave it as the only thing you use.

The Model Context Protocol has three layers, not one. Resources, Tools, and Prompts. Each of them solves a different problem, each of them costs a different amount of tokens, and each of them is supported unevenly across clients right now. On the same browser-automation task, a CLI-style setup that pre-loaded the full tool documentation into the context ran at 114,000 input tokens. The MCP-native setup, where the server abstracted the low-level details away, ran at 27,000. A 4.2x cut without changing the model or the underlying browser — pulled from the case study in the Context Engineering book this article is adapted from.

Below is the concrete framework I use to decide which layer a piece of context belongs in, why the three layers cost so differently, and what to do when your MCP client only supports the layers it feels like supporting.

The 3 layers, side by side

Layer	What it is	When to use	Cost profile
Resources	Read-only information sources the model can pull on demand (`file://`, `db://`, `api://`)	Static or slow-changing context the agent may need	Cheap. Pulled only when referenced.
Tools	Executable operations with side effects (`create_file`, `send_email`)	Actions the agent takes in the world	Moderate. Definition sits in context every call.
Prompts	Reusable, parameterized templates the client can invoke	Repeated flows: code review, ticket triage, incident response	Cheap. Server-side; the client fetches on request.

Skip any of these three and you either lose expressiveness or you overpay in tokens. Both mistakes are common.

Why 114K → 27K happened

The 114K number came from a CLI-style approach to browser automation: dump the full command-line documentation, error handling, and usage examples for every primitive into the context, so the LLM knows how to compose them. Something like this:

# CLI-style design (114K tokens per run)
context = "Available CLI commands:\n"
context += generate_cli_documentation()   # 30 primitives, each with full docs
context += generate_cli_examples()        # example usage for each
context += generate_error_handling_docs() # every error code + retry strategy
# task itself: ~8K tokens sitting on top of ~106K of tooling documentation

The 27K number came from an MCP-native rewrite where the server hides those details behind a small set of high-level tools. The chapter this article adapts frames it as roughly this:

# MCP-native design (27K tokens per run)
tools = [
    "playwright_navigate(url)",
    "playwright_click(selector)",
    "playwright_type(selector, text)",
    "playwright_screenshot()",
    "playwright_extract_text(selector)",
]
# server holds the CLI-level docs, retry policies, and error mapping internally

The book credits three mechanisms for the drop:

Abstraction level. CLI-style is low-level detail. MCP is a high-level abstraction. The client sends intent; the server figures out how to execute.
Separation of context. The MCP server holds implementation details. The client only sees what it needs to decide the next action.
Dynamic resolution. MCP resolves details when the tool actually runs. CLI-style has to describe everything up front, in case it is needed.

Where the three MCP layers come in is that each of them is one of those separation strategies. Tools are the intent layer. Resources are how the "docs the agent occasionally needs" get moved off the always-loaded context. Prompts are how the "repeat this whole flow again" logic gets moved onto the server. Together they are what makes the 4.2x abstraction gain possible in the first place.

The one-line takeaway from the chapter: your token bill is the description of the tools, not the tools themselves.

Which layer does a piece of context belong in?

The decision is not that hard once you have a rule. Here is the one I use.

Ask: does the agent need to do something, read something, or repeat something?

Do something → Tool. Side effects, mutations, external calls.
Read something → Resource. Read-only, addressable by URI, pulled on demand.
Repeat something → Prompt. Templates the client can invoke by name.

If you feel yourself dumping a full API doc into a tool's description field, stop. That is a Resource. If you feel yourself teaching the model the same 6-step recipe over and over in the system prompt, stop. That is a Prompt.

Tools are the loud, expensive layer because their definitions live in the context on every call. Resources and Prompts are the quiet, cheap layers because their content is fetched only when the agent decides it needs them. Most teams over-invest in Tools and under-invest in Resources and Prompts, then wonder why their agents feel expensive.

What your client actually supports right now (September 2026)

Here is the part nobody warns you about. The MCP spec has three layers. The clients do not implement all three uniformly.

Claude Desktop supports Resources, Tools, and Prompts. Prompts appear in the slash-menu inside the chat.
Cursor supports Tools well. Resource support has been improving through 2026 but is uneven. Prompt support arrived in a recent update and is now working end-to-end in the mcp.json config.
ChatGPT has been rolling out MCP since 2025 (Developer Mode beta shipped September 2025), with the Enterprise and Business rollout continuing into 2026. Tool support is solid. Resources and Prompts are still shipping.
Custom SDK clients (via the Python or TypeScript SDK) implement whatever you write. The floor is Tools; adding Resources and Prompts is trivial in the SDK and worth doing.

The practical takeaway is unglamorous. Even if your server exposes all three layers, some fraction of your users are on a client that only invokes Tools. Build your server so that a Tools-only client still gets meaningful behavior, and Resources and Prompts stack on top for the clients that can use them. Do not gate the core action on a Prompt the client will never call.

The July 28, 2026 spec revision helped here: capability discovery is now handled through the server/discover method, so a well-behaved client can at least ask what the server offers instead of assuming. Whether it actually uses what the server offers is a different question.

The design mistake I keep watching people make

The most common failure mode is not "picked the wrong layer." It is "put everything in Tools because Tools is the layer they understood first." I did this on my first two servers before I noticed the pattern. It is a rite of passage I would rather you skip.

Symptoms:

Tool descriptions running 500+ words each.
30+ tools on the surface.
System prompt contains "when using tool X, remember to first check Y" instructions.
Token cost per turn is embarrassing.

The fix is almost always the same shape: collapse related low-level tools into an intent tool, move the docs the agent occasionally needs into Resources, and lift the repeated flows into Prompts. The MCP server picks up the orchestration; the agent stops paying for it in tokens.

The design principle behind this is the one line from the MCP guide I keep rereading: the description of the tool is context, and context has a price.

What to do this week

If you already run an MCP server, three concrete actions.

Count the tokens your tool definitions consume on a typical run. If they are more than 20% of the total input budget, you have room to compress.
Identify one thing you currently put in the system prompt or in a tool description that is really reference material. Move it to a Resource this week.
Identify one multi-step flow you invoke often. Lift it to a Prompt so it lives on the server instead of being rebuilt in the client each time.

None of these three moves require rewriting your agent. They are configuration changes on the MCP server side, and the client will start benefiting the next time it connects.

If you want the longer walkthrough that this article is based on, my Zenn book Context Engineering: Turn LLMs From Liars Into Experts has a full chapter on the MCP three-layer split, including the code snippets for the Playwright server refactor that got me from 114K to 27K.

References

Model Context Protocol — Official spec and getting started
Maxim AI — What Is Model Context Protocol (MCP)? A 2026 Guide
Descope — What Is MCP and How It Works
Prompt Architects — How to Use MCP to Manage Prompts Inside Cursor & Claude Desktop

Claude Code Hooks vs Skills vs Subagents: Three Ways to Extend the Agent, and When Each Backfires

Ken Imoto — Sun, 13 Sep 2026 13:00:01 +0000

The first time I tried to make Claude Code "always run the linter," I wrote a Skill. It worked most of the time. The rest of the time, Claude decided the task didn't need linting and skipped it. I spent an embarrassing afternoon tightening the Skill's description before I understood the actual problem: I had picked the wrong mechanism. "Always do X" is a Hook. "Decide whether to do X" is a Skill. They are not interchangeable, and the docs at the time were happy to let me confuse them.

Claude Code now ships four ways to extend the agent: Hooks, Skills, Subagents, and the Agent SDK. They overlap enough to look like alternatives and differ enough that using one where another belongs costs you reliability, tokens, or both. I've built real harnesses on the first three, and read the SDK docs closely enough to know where it takes over. Here is how each one actually fires, what it costs you in context, and the specific way each one bites back.

The one-line mental model

Before the table, the version I tell teammates:

Hooks are deterministic. They fire on an event whether Claude likes it or not.
Skills are probabilistic. Claude reads a description and decides to pull them in.
Subagents are isolation. They run in their own context window and hand you back a summary.
The Agent SDK is you taking the wheel: the same agent loop, running in your own process.

If you remember nothing else: Hooks remove a decision from the model, Skills add one, Subagents quarantine one.

The comparison

Mechanism	How it fires	Token cost	Best for	When it backfires
Hooks	Lifecycle events (`PreToolUse`, `Stop`, etc.) defined in `settings.json`. Not the model's choice.	Runs as a shell command outside the context window. Near zero, unless you deliberately inject stdout.	Non-negotiable rules: format on write, block secret files, run tests before stop.	A hook exiting `2` blocks the action. A bad `Stop` hook loops the agent forever.
Skills	Claude matches your prompt against the Skill's `description`, or you type `/name`.	Progressive disclosure: only `name`+`description` (a few dozen tokens) loaded per Skill at session start; body loads on invoke.	Reusable procedures Claude should choose when relevant: a review workflow, a release-notes drafter.	A bloated `description` taxes every turn and misroutes. The model can decline to fire at all.
Subagents	Auto-delegated when a task matches the agent's `description`, or via the `Agent` tool (the old `Task`).	Runs in its own context window. Only the final message returns to the parent.	Quarantining noisy work: codebase-wide greps, log trawls, parallel exploration.	It can't ask you follow-ups mid-task, and the summary drops the detail you needed.
Agent SDK	Programmatic. You call `query()` from TypeScript or Python and drive the loop yourself.	A separate program in your own process. Context is whatever your code feeds it.	Productizing the agent: CI bots, backend services, anything outside the CLI.	You own retries, sandboxing, session state, and cost. Nothing is automatic anymore.

Hooks: the mechanism that removes the model's vote

A Hook is a shell command wired to a lifecycle event in settings.json. The model does not get a say. This is the whole point and the whole danger.

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [{ "type": "command", "command": "prettier --write \"$CLAUDE_FILE_PATHS\"" }]
      }
    ]
  }
}

The 2026 event surface is much wider than the original set. Beyond the familiar PreToolUse, PostToolUse, UserPromptSubmit, SessionStart, Stop, and Notification, there are now session events like SessionEnd, agent events like SubagentStart/SubagentStop, and context events like PreCompact/PostCompact (hooks reference).

The footgun is the exit code. Exit 0 is success. Exit 2 is a blocking error: stderr goes back to Claude and the action is rejected. On a PreToolUse hook that's a feature: you can stop Claude from touching .env. On a Stop or SubagentStop hook, exit 2 means "you are not allowed to stop, keep going." I once wrote a Stop hook that checked for uncommitted changes and exited 2 if it found any. The agent dutifully refused to stop, tried again, still had uncommitted changes, refused to stop again. I had built a loop with my own hands. The lesson: a Hook that can block is a Hook that can deadlock. Test the exit-2 path before you trust it.

Because Hooks run with your shell privileges and outside the context window, they cost almost nothing in tokens. That's why "always run the linter" belongs here and not in a Skill. You are not asking Claude to remember; you are removing the choice.

Skills: the mechanism that adds the model's vote

A Skill is a SKILL.md file with frontmatter. Claude reads the description of every installed Skill at session start and decides, per turn, whether your prompt warrants pulling one in. You can also force it with /name.

The mechanic that makes Skills scale is progressive disclosure. Three tiers load at three different times:

Always loaded: the name and description only, a few dozen tokens per Skill. This is how one agent can know about hundreds of Skills without drowning.
Loaded on invoke: the SKILL.md body. Anthropic's own guidance keeps the body under ~500 lines (skill-development SKILL.md).
Loaded only when referenced: bundled scripts and detail files, zero tokens until Claude reaches for them.

So the failure mode is specific and quiet. Every Skill's description sits in context on every turn. Write a paragraph there and you've levied a permanent tax on the whole session, and you've made the routing decision harder, so the model misfires more often. My original "always lint" Skill failed because I was asking a probabilistic router to behave deterministically. The router did its job: it routed. Sometimes away from me.

Skills are right when you genuinely want Claude to choose. A code-review workflow, a "draft release notes from the git log" procedure, a triage routine. Things that should happen when relevant, judged by the model. Write the description like a routing key, not a summary (Agent Skills overview).

Subagents: the mechanism that quarantines context

A Subagent is a Markdown file in .claude/agents/ with frontmatter, and its body becomes that agent's system prompt:

---
name: codebase-explorer
description: Search the codebase for where a symbol is defined and used. Use proactively before refactors.
tools: Read, Glob, Grep
model: sonnet
---

You locate code. Report file paths and line numbers, not full file dumps.

The defining property: it runs in its own context window. Its system prompt, its tool calls, every grep result and log line it reads, none of that enters your main conversation. Only the final message comes back (custom subagents). This is the single best tool for a task that would otherwise flood your context: "find every caller of this function across 400 files." Run it in a Subagent and you get back a clean list instead of 400 files of noise.

Two backfires bit me here. First, a Subagent can't stop to ask you a question. It runs to completion and reports once. If it hits an ambiguous decision halfway through, it guesses, and you find out from the summary. Second, that summary is lossy by design. The detail lived in the isolated window and is gone unless the agent thought to surface it. I've had a Subagent confidently report "no usages found" when it had quietly searched the wrong directory. The fix is to write the agent's instructions to return evidence (paths, counts), not just conclusions.

One 2026 gotcha worth flagging: permission modes are inherited from the parent and can override the Subagent's own frontmatter. If your main session is running with relaxed permissions, the Subagent's careful tools allowlist may not save you. Treat the tool list as scoping, not a security boundary.

Agent SDK: the mechanism that hands you the keys

When the agent needs to live outside the terminal, in CI, a backend service, a scheduled job, you reach for the Agent SDK. It was renamed from the Claude Code SDK because the same loop powers far more than coding (migration guide). It ships for both TypeScript (@anthropic-ai/claude-agent-sdk) and Python (claude-agent-sdk).

from claude_agent_sdk import query, ClaudeAgentOptions

async for message in query(
    prompt="Find and fix the failing test in auth.py",
    options=ClaudeAgentOptions(allowed_tools=["Read", "Edit", "Bash"]),
):
    print(message)

You get the same agent loop, built-in tools, hooks, subagents, and MCP that power the CLI, but now running in your process. The backfire is everything the CLI used to do for free: retries, sandboxing, session persistence, and cost control are now your problem. One auth constraint to know before you build a product on it: third-party products on the Agent SDK can't use claude.ai subscription login, you authenticate with an API key (directly or via Bedrock/Vertex). Prototype on the SDK, then decide whether a hosted option fits production.

How I actually choose

The question I ask, in order:

Must this happen every time, no exceptions? Hook.
Should Claude decide when it's relevant? Skill.
Is this going to dump a pile of junk into my context? Subagent.
Does this need to run without me in the loop? Agent SDK.

Most of my mistakes came from answering question 2 when the honest answer was question 1. "Always" and "when relevant" feel similar when you're typing the config. They are opposites at runtime.

Takeaways

Hooks remove the model's choice. Use them for non-negotiable rules. Watch the exit-2 path: a blocking Stop hook can loop your agent.
Skills add a choice. Progressive disclosure keeps them cheap, but every description is taxed on every turn. Don't use a Skill to enforce something that must always happen.
Subagents quarantine context. Best for noisy exploration. They can't ask follow-ups and the summary is lossy, so make them return evidence.
The Agent SDK hands you the keys. Same loop, your process, your operational burden.
The biggest reliability win isn't picking the "best" mechanism. It's noticing when "always" is masquerading as "when relevant."

If you've been wiring Claude Code config long enough to feel the shape of a coherent harness, where Hooks, Skills, Subagents, and CLAUDE.md all sit in one place and reinforce each other, that's the book I wrote.

👉 Claude Code Mastery