Our framework docs, three models, isolated agents: 99 seconds, 50 minutes, and a timeout

A week ago we wrote about an eval for the agent docs that ship inside our framework package. A fresh coding agent passed all four tasks, but the run had a hole: the agents inherited our own Claude Code config, so nothing proved that the package docs alone were enough. The fix was to run each agent in isolation. That run is done now. We also gave the first task to two local models to see how far they get on a 48 GB Mac.

Kumiko is our TypeScript framework for multi-tenant apps on Bun, Hono and Postgres. Since 0.364.0 the npm package contains an AGENTS.md plus generated indexes of features, widgets and sample apps. The eval checks whether an agent that has never seen the framework source can build features from that.

The setup

Each task gets a fresh app scaffolded from the published packages, version 0.393.0. The agent runs as claude -p (Claude Code 2.1.310) in the project directory with CLAUDE_CONFIG_DIR pointing at an empty temp directory, so none of our skills, rules or memory leak in. It sees the app, its node_modules and the task text. Afterwards a grader copies in a hidden acceptance test and runs eight checks: typecheck, lint, unit tests, boot, schema validation, the acceptance test against a real Postgres, and two guard runs.

The four tasks: an entity with CRUD and a list screen, custom write handlers, an overview screen with stats, and wiring in the bundled tags feature.

First, the grader was broken

The first isolated Sonnet run failed every task. The agents' code was fine. Our grader had rotted.

The acceptance tests imported setupAppTestStack and seedTenant from @cosmicdrift/kumiko-testing. Those helpers moved to @cosmicdrift/kumiko-dev-server/testing some releases ago, and the templates never followed. The guard check ran bunx kumiko-guards@0.393.0, which installs the guards package on its own. That package imports @cosmicdrift/kumiko-types without declaring it, so it crashed before checking anything. Inside an app the import resolves by luck, because the framework pulls the types package in.

We fixed the templates, made the grader run the guards the project already has installed, and declared the missing dependency (kumiko-framework#4074). All results below come from that fixed grader.

The funny part came from Sonnet's own report on task 02. Each agent ends with a list of things the docs didn't tell it. Sonnet wrote that AGENTS.md names the wrong package for setupAppTestStack and seedTenant. It was right: our docs had the same stale import as our grader. The agent noticed, copied the working import from the scaffolded test, and moved on. That line is fixed in kumiko-framework#4075.

Sonnet 5.5: four out of four

Task	Checks	Wall time	Responses / tool calls	Median response	p90 response	Cost
01 entity CRUD	8/8	99 s	13 / 12	4.0 s	7.9 s	$0.24
02 custom handlers	8/8	60 s	8 / 7	3.6 s	12.2 s	$0.22
03 overview screen	8/8	35 s	4 / 3	5.8 s	15.3 s	$0.15
04 bundled tags	8/8	73 s	7 / 6	6.3 s	21.4 s	$0.20

"Response time" is measured from the moment the agent gets a prompt or a tool result to the last block of the model's answer, taken from the stream timestamps. All four tasks took under five minutes and $0.82 in total. The grader also scans the diff for patterns our rules forbid. It found one hit, an any inside a string literal in task 04, which is a false positive.

So the claim from last week holds with isolation: the package docs are enough for a strong model to finish these four tasks.

Two local models on task 01

The local runs used the same Claude Code client against a local Anthropic-compatible endpoint. The machine is an M4 Pro with 48 GB of unified memory, of which about 37 GiB is available to the GPU.

Strata 0.2.6 with qwen3.8-flash-next at Q2_0 and 128K context. The weights are 66 GB, so the model does not fit in GPU memory at all. We wanted to see whether it runs anyway. It did, at about 4.5 tokens per second for output and 9 tokens per second for reading the prompt. Every request carried 39k to 43k tokens of context, and almost all of that came from the prompt cache; without the cache this run would not have finished in any useful time.

It passed. All eight checks green, no hits in the rule scan, in 50 minutes.

Qwen3.5 9B on Ollama 0.31.2 with 64K context. At 6.6 GB it fits entirely on the GPU and answers faster per turn. We gave it a 60-minute cap. It hit the cap with two of eight checks green, and those two were the guards, which pass on an app that barely changed. The app never booted ("Export named 'contactsFeature' not found"). On the way the model wrote feature code into a .gitignore file and kept rewriting the same files, and Claude Code compacted its context twice.

Model	Checks	Wall time	Responses / tool calls	First response	Median response	p90 response
Sonnet 5.5 (API)	8/8	99 s	13 / 12	4.5 s	4.0 s	7.9 s
Strata Q2_0, 66 GB	8/8	50 min	49 / 68	276 s	36.8 s	141 s
Qwen3.5 9B, Ollama	2/8	60 min, stopped	82 / 81	67 s	26.5 s	66.9 s

The first response is the slowest one for the local models because the whole system prompt and tool list have to be read once. For Strata that took four and a half minutes; its slowest answer took five.

What the numbers don't say

Every model ran each task once. One pass or one failure is an anecdote.

The local runs had output capped at 4,096 tokens per response. Sonnet had no cap. The context windows differed too: 128K for Strata, 64K for the 9B.

The local models only got task 01, the simplest one. A 50-minute pass on that says little about tasks 02 to 04.

Strata's speed depends heavily on the prompt cache. A task that invalidates the cache more often would be much slower.

What the docs still miss

Each agent lists what it had to guess. These came up in the isolated runs:

• personalData is described for writes, but not what happens on reads or sorting. Sonnet asked whether a PII field can be a default sort.
• How a screen gets registered in client.tsx isn't documented.
• What a create result contains, and that a ne filter excludes NULL rows.
• Which error class fits "this is already done".
• How to wire a tags section into an entity edit screen, and whether entityName must be set. The tags recipe also uses the framework's internal test setup instead of the app's.
• Strata asked which find level fits and which translation key a row action label uses.

None of these blocked a pass, which is the uncomfortable part. A strong model fills these gaps by reading compiled code or guessing well, and the eval stays green.

What we take from it

The isolated run confirms the docs work for Sonnet. It also showed that our grader and our docs had drifted from the code in the same place, and the agent caught it before we did. An eval that nobody runs for a week rots like any other code.

For local models, a heavily quantized model that doesn't fit in memory finished task 01 correctly, and a small model that fits easily didn't. In this one run, model size mattered more than speed.

If you've run coding agents on local models against a real codebase: did a bigger model at 2 bits beat a smaller one at 4 or 8 bits for you, or was our run a fluke?