A fresh coding agent passed all four tasks against our framework docs. Here's what that result doesn't prove

Kumiko is our TypeScript framework for multi-tenant apps on Bun, Hono and Postgres. More and more of the code written against it comes from coding agents, and until last week those agents saw very little of it. An agent working in an app sees the app and its node_modules. Our framework package shipped dist, a README and a license. The docs site exists, but it's outside what an agent sees in an app, and the instructions written for agents lived in skills inside our own workspace, so only our own sessions ever read them.

So we moved the explanation into the package and then checked whether an agent that has never seen the framework source can work from it.

What ships in the package now

Since 0.364.0, @cosmicdrift/kumiko-framework contains an AGENTS.md, today about 350 lines. It explains the model (features, entities, handlers, screens, events, tenant scoping), lists rules in the form "never this, do that", and has a table of "I need X, use Y". Next to it sits an agents/ folder that the build generates from the source: an index of every bundled feature from the feature manifest, an index of every UI widget export, and one file per tested sample app with its README, source and tests. A drift guard fails CI when the generated files no longer match the code.

A new app from create-kumiko-app gets a short AGENTS.md of its own for other agents and a CLAUDE.md that imports the package file:

@node_modules/@cosmicdrift/kumiko-framework/AGENTS.md

The docs always match the installed version and work offline. Boot errors and guard messages also carry a docsUrl that points at a page for that error code.

The eval

We wrote four tasks of the kind a real app needs:

1. An entity with a list and a form.
2. A write handler and a query handler with integration tests (complete a task, count open tasks per tenant).
3. An overview screen built from existing widgets.
4. Mounting the bundled tags feature in an app.

Each task got its own fresh scaffold, installed from npm into a directory outside any workspace. Inside a workspace, bun resolves workspace links instead of the npm tarball, and the agent reads our source instead of the package. The prepare script refuses to run inside a directory that has a package.json or bun.lock above it.

Each project got a fresh agent (Sonnet) with a prompt that contained the environment, the task text and the rules: work only in the project, read only the project and its node_modules, no skills, no memory, no web search. At the end it had to report in 15 lines what it built, which docs it read and what was missing.

We didn't grade on that report. A script copies a hidden acceptance test into the project that the agent never saw, then runs typecheck, lint, unit tests, boot validation, schema validation, the integration tests (the agent's own plus the hidden one) and our guards. It also scans the agent's diff for any, raw SQL writes, inline styles and a test helper we don't allow in integration tests.

Result: 4 of 4

All eight checks passed in every project, including the hidden acceptance tests. In task 2 the integration run had 13 tests: 8 written by the agent and 5 from the acceptance file.

Every task still showed a gap in the docs:

Task	What the agent couldn't find in the docs
Entity with list and form	The personal-data setting a write handler needs when it writes personal fields. It found it through the boot error.
Handlers with tests	How to count. list with totalCount and a filter was only in the .d.ts files.
Overview screen	Any example of a dashboard screen. It worked out the data contract from the compiled renderer.
Mount tags	That a newly mounted bundled feature needs a migration, and how to wire it into the client and the test stack.

The boot errors did their job in task 1. The agent made the mistake, the error named the fix, and it applied it. In the other three, the docs alone weren't enough, and the agents filled the gap by reading type declarations and compiled JavaScript in node_modules. The rules allowed that, and a human developer would do the same. For this eval we count it as a gap anyway, because the next agent might not look there.

Why 4 of 4 is weaker than it looks

The task texts fixed names, payloads and IDs, so the hidden tests could find what they test. An open request like "add a way to see how many tasks are left" is harder.

Task 3 passed on its declaration and its data. Nothing rendered a screen, so we don't know whether the title key resolves.

The bigger problem is how the agents ran. To save time, we started them as subagents from an existing Claude Code session instead of in a clean CLI with an empty config directory. A subagent inherits the session's user and workspace instructions, and ours state the same rules the review checks: no any, the real test stack instead of a test dispatcher, no direct writes to event tables. Our results table has a column for rule breaks, and it was empty for all four runs. That says nothing about whether the package docs teach those rules. Only the columns for which docs an agent read and what it missed measure the package docs on their own.

We checked the transcripts for paths outside the project. None of the four agents touched our source tree or our config folder, and none used a skill or a web search.

What the eval found that wasn't documentation

The fresh scaffolds were broken in ways we never noticed, because we never start from a fresh scaffold. The prepare script had to patch some of this before an agent could start (version pins, the .gitignore, the Biome config, a random master key and admin password in .env), and none of those patches help with the tasks:

• The scaffold wrote no .gitignore. Biome is configured to read it, so bun run lint failed in a brand new project.
• Biome was pinned with a caret range that now resolves to a newer minor version. The new project was lint-red on its first run.
• The generated production entry point seeded an admin user with a fixed password. Since 0.365.0 it requires DEMO_ADMIN_PASSWORD (at least 12 characters) in the env schema, and the boot stops before seeding when it is missing.

Upgrading the framework doesn't rewrite your bin/main.ts. If you scaffolded an app with 0.364.0 or earlier and booted it in production mode, change that admin's password and add the env variable by hand.

One of our own examples taught the wrong thing. The counter:active query in the custom-handlers sample loaded a page with list and filtered it in JavaScript. That only ever sees the first page, and it's the example an agent would copy when it needs to count. It now filters in the query and asks for totalCount.

While writing the error pages we also found that every docsUrl led to a 404. The framework built /errors/<code>, and the docs site served /en/errors/<code>/. The site now redirects, and a check fails the docs build when an error code has no page.

Next step

The run the eval was designed for is still open: one agent per project, started in the project directory with an empty Claude Code config directory, so nothing from our setup leaks in. The README in eval/agent-docs/ describes it, and that run is what tells us whether the rules in AGENTS.md reach an agent that has only the package.

The tasks, the hidden acceptance tests and the runner are in the public repo under eval/agent-docs/.

How do you grade an eval like this? We count "had to read compiled code" as a docs gap even when the task passes. We're not sure that's the right line, or whether it should fail the task.