DEV Community: Marc

Our framework docs, three models, isolated agents: 99 seconds, 50 minutes, and a timeout

Marc — Mon, 26 Oct 2026 19:53:45 +0000

A week ago we wrote about an eval for the agent docs that ship inside our framework package. A fresh coding agent passed all four tasks, but the run had a hole: the agents inherited our own Claude Code config, so nothing proved that the package docs alone were enough. The fix was to run each agent in isolation. That run is done now. We also gave the first task to two local models to see how far they get on a 48 GB Mac.

Kumiko is our TypeScript framework for multi-tenant apps on Bun, Hono and Postgres. Since 0.364.0 the npm package contains an AGENTS.md plus generated indexes of features, widgets and sample apps. The eval checks whether an agent that has never seen the framework source can build features from that.

The setup

Each task gets a fresh app scaffolded from the published packages, version 0.393.0. The agent runs as claude -p (Claude Code 2.1.310) in the project directory with CLAUDE_CONFIG_DIR pointing at an empty temp directory, so none of our skills, rules or memory leak in. It sees the app, its node_modules and the task text. Afterwards a grader copies in a hidden acceptance test and runs eight checks: typecheck, lint, unit tests, boot, schema validation, the acceptance test against a real Postgres, and two guard runs.

The four tasks: an entity with CRUD and a list screen, custom write handlers, an overview screen with stats, and wiring in the bundled tags feature.

First, the grader was broken

The first isolated Sonnet run failed every task. The agents' code was fine. Our grader had rotted.

The acceptance tests imported setupAppTestStack and seedTenant from @cosmicdrift/kumiko-testing. Those helpers moved to @cosmicdrift/kumiko-dev-server/testing some releases ago, and the templates never followed. The guard check ran bunx kumiko-guards@0.393.0, which installs the guards package on its own. That package imports @cosmicdrift/kumiko-types without declaring it, so it crashed before checking anything. Inside an app the import resolves by luck, because the framework pulls the types package in.

We fixed the templates, made the grader run the guards the project already has installed, and declared the missing dependency (kumiko-framework#4074). All results below come from that fixed grader.

The funny part came from Sonnet's own report on task 02. Each agent ends with a list of things the docs didn't tell it. Sonnet wrote that AGENTS.md names the wrong package for setupAppTestStack and seedTenant. It was right: our docs had the same stale import as our grader. The agent noticed, copied the working import from the scaffolded test, and moved on. That line is fixed in kumiko-framework#4075.

Sonnet 5.5: four out of four

Task	Checks	Wall time	Responses / tool calls	Median response	p90 response	Cost
01 entity CRUD	8/8	99 s	13 / 12	4.0 s	7.9 s	$0.24
02 custom handlers	8/8	60 s	8 / 7	3.6 s	12.2 s	$0.22
03 overview screen	8/8	35 s	4 / 3	5.8 s	15.3 s	$0.15
04 bundled tags	8/8	73 s	7 / 6	6.3 s	21.4 s	$0.20

"Response time" is measured from the moment the agent gets a prompt or a tool result to the last block of the model's answer, taken from the stream timestamps. All four tasks took under five minutes and $0.82 in total. The grader also scans the diff for patterns our rules forbid. It found one hit, an any inside a string literal in task 04, which is a false positive.

So the claim from last week holds with isolation: the package docs are enough for a strong model to finish these four tasks.

Two local models on task 01

The local runs used the same Claude Code client against a local Anthropic-compatible endpoint. The machine is an M4 Pro with 48 GB of unified memory, of which about 37 GiB is available to the GPU.

Strata 0.2.6 with qwen3.8-flash-next at Q2_0 and 128K context. The weights are 66 GB, so the model does not fit in GPU memory at all. We wanted to see whether it runs anyway. It did, at about 4.5 tokens per second for output and 9 tokens per second for reading the prompt. Every request carried 39k to 43k tokens of context, and almost all of that came from the prompt cache; without the cache this run would not have finished in any useful time.

It passed. All eight checks green, no hits in the rule scan, in 50 minutes.

Qwen3.5 9B on Ollama 0.31.2 with 64K context. At 6.6 GB it fits entirely on the GPU and answers faster per turn. We gave it a 60-minute cap. It hit the cap with two of eight checks green, and those two were the guards, which pass on an app that barely changed. The app never booted ("Export named 'contactsFeature' not found"). On the way the model wrote feature code into a .gitignore file and kept rewriting the same files, and Claude Code compacted its context twice.

Model	Checks	Wall time	Responses / tool calls	First response	Median response	p90 response
Sonnet 5.5 (API)	8/8	99 s	13 / 12	4.5 s	4.0 s	7.9 s
Strata Q2_0, 66 GB	8/8	50 min	49 / 68	276 s	36.8 s	141 s
Qwen3.5 9B, Ollama	2/8	60 min, stopped	82 / 81	67 s	26.5 s	66.9 s

The first response is the slowest one for the local models because the whole system prompt and tool list have to be read once. For Strata that took four and a half minutes; its slowest answer took five.

What the numbers don't say

Every model ran each task once. One pass or one failure is an anecdote.

The local runs had output capped at 4,096 tokens per response. Sonnet had no cap. The context windows differed too: 128K for Strata, 64K for the 9B.

The local models only got task 01, the simplest one. A 50-minute pass on that says little about tasks 02 to 04.

Strata's speed depends heavily on the prompt cache. A task that invalidates the cache more often would be much slower.

What the docs still miss

Each agent lists what it had to guess. These came up in the isolated runs:

personalData is described for writes, but not what happens on reads or sorting. Sonnet asked whether a PII field can be a default sort.
How a screen gets registered in client.tsx isn't documented.
What a create result contains, and that a ne filter excludes NULL rows.
Which error class fits "this is already done".
How to wire a tags section into an entity edit screen, and whether entityName must be set. The tags recipe also uses the framework's internal test setup instead of the app's.
Strata asked which find level fits and which translation key a row action label uses.

None of these blocked a pass, which is the uncomfortable part. A strong model fills these gaps by reading compiled code or guessing well, and the eval stays green.

What we take from it

The isolated run confirms the docs work for Sonnet. It also showed that our grader and our docs had drifted from the code in the same place, and the agent caught it before we did. An eval that nobody runs for a week rots like any other code.

For local models, a heavily quantized model that doesn't fit in memory finished task 01 correctly, and a small model that fits easily didn't. In this one run, model size mattered more than speed.

If you've run coding agents on local models against a real codebase: did a bigger model at 2 bits beat a smaller one at 4 or 8 bits for you, or was our run a fluke?

Changing a mail template used to need a deploy. We built an editor that refuses to publish a broken one

Marc — Sat, 24 Oct 2026 20:50:03 +0000

Until recently, changing the wording of a password-reset mail in our framework meant editing a TypeScript file and shipping a release. The same went for the PDF a rental app sends as an invoice. Nobody on the operations side could touch either one.

Looking at why, we found two mail systems that did not know about each other. One stored the mail as a JSON list of sections (text, button) in code, with the branding hardcoded next to it. The other stored a Markdown document in the database and rendered it through MJML, and exactly one app used it. PDFs were a third thing: HTML built in code and printed with Puppeteer. The tenant's logo and colors lived in two places, the static mail branding in app code and the branding tokens per tenant. The mail message type had no concept of attachments or inline images at all.

One model for mail and PDF

Every mail and every PDF is now a template: a slug, a locale, a document, and a small JSON meta object. The document is structured content that tiptap edits in the browser. The server rebuilds it through a sanitizer, so only known node types survive a save. Meta is validated per kind: a mail needs a subject and may have preview text, a layout, a category and a sender name. A PDF has page format, margins and a file name. Images and files are linked resources, uploaded once and referenced by key.

Layout is a separate template. A mail layout (MJML) and a PDF letterhead each have exactly one required slot for the content, and they read the tenant's branding tokens instead of carrying their own copy. Changing the layout changes every mail that uses it, without touching any sender.

Features no longer send sections. They declare which variables a mail needs and seed a default template at boot. This is the declaration for the waitlist notice in the framework:

export const WAITLIST_ADMIN_NOTICE_VARIABLES: TemplateVariables = {
  name: templateVariable.text({ required: true }),
  details: templateVariable.list(
    { text: templateVariable.text({ required: true }) },
    { required: true },
  ),
};

The variable types are text, link, money, date and list (the schema also knows rich text and percent). Each one carries a label and an example value, and the example is what the preview fills in.

Scopes and the four-step lookup

A template can exist at two scopes: the system default that the feature ships, and a tenant customisation. When a handler asks for a template, the resolver walks four steps: the tenant in the requested locale, the system in the requested locale, the tenant in the fallback locale, the system in the fallback locale. A tenant overrides a default by having its own row, and no application code changes.

Who may edit sits outside the model. By default the platform admin edits the system templates, and an app can open individual templates to its tenant admins. Mails that must stay as shipped can be marked system only, and tenants neither see nor override them.

The editor

The overview lists mails, PDFs and layouts in one table. Each row shows where the template comes from (system or tenant customisation), which languages exist as filled chips, the status with its version, and which feature uses it. The usage column comes from the feature registry, so it is read from the code that sends the mail and nobody types it in.

The mail editor has the placeholder rail on the left with type, required marker and a check for whether the placeholder is used. The middle holds subject, preview text and the content with placeholders as chips and blocks for buttons, images and repeated rows. The right side has layout, sender, category and attachments. In the screenshot, a required placeholder, the due date, appears nowhere in the subject, preview text or content. The footer says so and the publish button is disabled. Saving a draft still works.

That rule is enforced on the server and not only in the editor. Saving rejects unknown placeholders. Saving an active template and publishing both check that every required variable is used. The editor only shows what the server would say.

A draft over an active template does not replace it. The active version keeps being served while the edit sits beside it, and publish switches over. The first version got this wrong: a draft saved over an active tenant template overwrote the same row, the resolver fell back to the system text, and the tenant's active wording was gone. The draft now lives in its own column, and publish promotes it.

PDFs

PDFs use the same document model, rendered through Puppeteer in a separate service that runs Chromium behind an egress guard. The preview is the real render. It shows the letterhead from the layout, the repeating rows from a list variable and the page count, and the footer line reports the last test render, here 2 pages, 67 KB, 2.3 seconds. Page format, margins, the file name pattern and the PDF metadata sit in the side panel. A generated PDF is stored with the template id and version it came from, so you can say later which wording a customer received.

Without a running Puppeteer service the PDF editor is locked, because a preview that cannot render would only mislead.

What is not in this release

Everything above is in the open framework, including the editors, the preview and the test mail. The AI assistant that edits templates from a sentence ("put the logo in the sign-up mail") is a separate piece that sits in our enterprise layer. It reads a template as a list of blocks and patches the draft through structured operations, and it never writes raw HTML. It shows the result as a preview first and saves it as a draft. Publishing stays a decision of the person using it. The same validation applies, so a suggestion that removes a required placeholder is rejected. Previewing a template with a real record is part of that layer too. This post is about the open part, so we only mention the assistant here.

Also still missing: version history with restore, columns, PDFs without Puppeteer, and an editor control for conditional blocks. The renderer already shows or hides a paragraph or table row depending on whether a variable has a value; the editor just has no button for it yet.

If you let non-developers edit transactional mail, what did you put between them and a broken invoice mail? We block publishing on a missing required placeholder and keep the active version served. We would like to hear what else you check before a template goes live.

Our comment section redesigned our agent permissions. Here's what we still can't solve

Marc — Wed, 21 Oct 2026 07:08:14 +0000

In September I wrote about AI agents in our framework, Kumiko, and why a handler without a description never shows up in the agent's tool list: Undocumented handlers don't exist to the agent. Two readers, phongdesigns and anp2network, went through the post more carefully than I had, and the thread under it ran to eleven comments over three days.

Three changes to how our agent decides what it may do came out of that thread. We turned down two of the ideas, and there are three questions we haven't answered. This post covers all of it, because the open questions are the part I'd like help with.

The setup in two sentences

Every write handler in a Kumiko app can carry an agent risk (low, mid or high), and the user can set mid-risk tools to "always allow" while high-risk tools ask every time. Back then, the risk was whatever the handler declared, plus two tests that pinned the risk of specific handlers by name.

Change 1: the risk comes from what the handler does

anp2network pointed out that pinning by name doesn't catch the hard delete someone adds in six months. It defaults to mid, and nothing goes red. Their suggestion was to watch what a handler actually does and compare that with what it declares.

That fit our code better than I expected. A hard delete in Kumiko doesn't go through the normal event append. It calls a separate forget operation on the event store, so the framework can see it. Since 0.322.0, a handler that runs forget, or a hard delete on an entity without soft delete, has to resolve to high. Otherwise the executor denies the call before it reads anything. The check looks at the handler the caller dispatched directly, so a mid-risk handler can't get around it by delegating to a high-risk one internally. Standard delete handlers on those entities default to high, and declaring one lower fails at boot.

My follow-up comment listed after-commit hooks, jobs and event consumers as a gap. Since then, writes dispatched inside those contexts go through the same check, with a short list of deliberate exceptions such as cron cleanups and pipelines started by a high-risk handler.

Change 2: the tool list and the dispatch table are tested against each other

I had claimed that the model can't call a tool that isn't in its list, because the list and the dispatch table come from the same catalog. anp2network's reply was that this was only true because of how the code happened to be arranged, and no test checked it. Now a test asserts that every tool name has exactly one dispatch entry, and the other way round.

Change 3: some fields are dangerous to write because of who reads them

This one I didn't see coming. Our prompt store keeps every revision of a feature's system prompt. Editing a prompt was a high-risk action. Reverting to an older revision was mid, and a revert appends a copy of the old revision, which then becomes the live prompt. A user who set revert to "always allow" had given the agent a way to swap another feature's instructions without asking.

A floor on the handler can't see that, because on the storage side the revert is an ordinary, reversible write. anp2network put it this way: the risk is a property of the destination. A later run reads that row as instructions.

Since 0.324.0, a field can say so:

systemPrompt: createLongTextField({
  required: true,
  personal: false,
  reason: "is_business_data",
  readAsInstruction: true,
}),

Any create or update that writes such a field needs high on the directly dispatched handler, through a generic update handler as well. The prompt store's content field carries the flag, so revert is high now.

We also record how every agent write was approved: no approval needed, an "always" rule, or the user confirming it. "Always" isn't offered for high-risk tools, so the number of flagged writes approved by an "always" rule should be zero, and we can check that with a query.

What we turned down

anp2network noticed that a tool the user set to "never" vanishes from the model's view, so to the model "forbidden" looks exactly like "doesn't exist". The suggestion was a tombstone: keep the entry visible and have dispatch refuse it with its own reason. We didn't build it. "Never" is one of several filters that hide tools (roles, mode, an explicit deny list), so tombstones would either be inconsistent with the other filters or leak names those filters exist to hide. anp2network agreed with the reasoning. Instead, the agent's system prompt now tells it to say the action isn't available "rather than approximating it with a similar tool — a similar tool is not the same action."

The other suggestion was a test that counts the high-risk handlers, so every change to that set shows up in a diff. Once the risk comes from the effect, the count adds little: a new hard delete is high whether anyone updates a number or not.

Still open

1. Irreversible effects outside our database

The floor sees forget and hard deletes. It doesn't see a mail, a webhook or a payment. In our store, the write that queues a mail looks like any other reversible write, but the mail can't be unsent. Our webhooks now send an idempotency key, so the receiver can drop a redelivery, but that only protects against duplicates. Nothing decides whether the agent should trigger the first one without asking.

Our half idea is to treat outbound channels like flagged fields: the channel declares itself irreversible, and any handler that hands work to it resolves to high. We haven't built that. I'm not sure the line is right either, because a status mail to the user themselves and a payment to a third party would land in the same class.

2. Who reads a field as instructions

The readAsInstruction flag is set by hand next to the field, apart from the code that later puts the field into a prompt. If a new feature starts feeding an unflagged field into a system prompt, the flag set is stale and every check stays green. anp2network asked whether anything fails when that happens. Nothing does, and on the agent side the system prompt is still a plain string.

The flag also has holes on the write side. A hand-written domain event whose projection writes the flagged field skips the gate. The gate checks whether the payload contains the field, not whether the value changed, which we chose deliberately so it fails closed. Reverting a prompt also doesn't undo a run that already read the bad version.

Our half idea is a type. The system prompt parameter would only accept text that came from a flagged field or from code, so feeding it anything else wouldn't compile. I haven't checked whether that survives prompts assembled from several pieces.

3. Refusing instead of reaching for the neighbour

When the model can't find a tool for what the user wants, it should say so. The risk is that it picks the closest-sounding tool instead, maybe one the user set to "always allow". Right now the only defence is the sentence in the system prompt quoted above.

anp2network suggested a number: after a rejected call, record what the model calls next. In our loop, calling a tool that isn't in the list ends the run, so there's no "next" to count. The case we can't see at all is the model going straight to the neighbour without ever trying the missing tool.

phongdesigns added the near-duplicate problem: two writes described in almost the same words, and the model confuses them. Our manifest snapshot still records only handler names, so a reworded description doesn't even show up in review. I said in the thread that adding descriptions and risk to it was the next step, and it still is.

The only test I can think of for substitution is to give a real model pairs of similar tools and see where it goes wrong. That costs money on every run, and the result changes with the model.

How do you test that an agent refuses instead of improvising, without paying for a model call on every pull request?

A fresh coding agent passed all four tasks against our framework docs. Here's what that result doesn't prove

Marc — Mon, 19 Oct 2026 21:09:58 +0000

Kumiko is our TypeScript framework for multi-tenant apps on Bun, Hono and Postgres. More and more of the code written against it comes from coding agents, and until last week those agents saw very little of it. An agent working in an app sees the app and its node_modules. Our framework package shipped dist, a README and a license. The docs site exists, but it's outside what an agent sees in an app, and the instructions written for agents lived in skills inside our own workspace, so only our own sessions ever read them.

So we moved the explanation into the package and then checked whether an agent that has never seen the framework source can work from it.

What ships in the package now

Since 0.364.0, @cosmicdrift/kumiko-framework contains an AGENTS.md, today about 350 lines. It explains the model (features, entities, handlers, screens, events, tenant scoping), lists rules in the form "never this, do that", and has a table of "I need X, use Y". Next to it sits an agents/ folder that the build generates from the source: an index of every bundled feature from the feature manifest, an index of every UI widget export, and one file per tested sample app with its README, source and tests. A drift guard fails CI when the generated files no longer match the code.

A new app from create-kumiko-app gets a short AGENTS.md of its own for other agents and a CLAUDE.md that imports the package file:

@node_modules/@cosmicdrift/kumiko-framework/AGENTS.md

The docs always match the installed version and work offline. Boot errors and guard messages also carry a docsUrl that points at a page for that error code.

The eval

We wrote four tasks of the kind a real app needs:

An entity with a list and a form.
A write handler and a query handler with integration tests (complete a task, count open tasks per tenant).
An overview screen built from existing widgets.
Mounting the bundled tags feature in an app.

Each task got its own fresh scaffold, installed from npm into a directory outside any workspace. Inside a workspace, bun resolves workspace links instead of the npm tarball, and the agent reads our source instead of the package. The prepare script refuses to run inside a directory that has a package.json or bun.lock above it.

Each project got a fresh agent (Sonnet) with a prompt that contained the environment, the task text and the rules: work only in the project, read only the project and its node_modules, no skills, no memory, no web search. At the end it had to report in 15 lines what it built, which docs it read and what was missing.

We didn't grade on that report. A script copies a hidden acceptance test into the project that the agent never saw, then runs typecheck, lint, unit tests, boot validation, schema validation, the integration tests (the agent's own plus the hidden one) and our guards. It also scans the agent's diff for any, raw SQL writes, inline styles and a test helper we don't allow in integration tests.

Result: 4 of 4

All eight checks passed in every project, including the hidden acceptance tests. In task 2 the integration run had 13 tests: 8 written by the agent and 5 from the acceptance file.

Every task still showed a gap in the docs:

Task	What the agent couldn't find in the docs
Entity with list and form	The personal-data setting a write handler needs when it writes personal fields. It found it through the boot error.
Handlers with tests	How to count. `list` with `totalCount` and a filter was only in the `.d.ts` files.
Overview screen	Any example of a dashboard screen. It worked out the data contract from the compiled renderer.
Mount `tags`	That a newly mounted bundled feature needs a migration, and how to wire it into the client and the test stack.

The boot errors did their job in task 1. The agent made the mistake, the error named the fix, and it applied it. In the other three, the docs alone weren't enough, and the agents filled the gap by reading type declarations and compiled JavaScript in node_modules. The rules allowed that, and a human developer would do the same. For this eval we count it as a gap anyway, because the next agent might not look there.

Why 4 of 4 is weaker than it looks

The task texts fixed names, payloads and IDs, so the hidden tests could find what they test. An open request like "add a way to see how many tasks are left" is harder.

Task 3 passed on its declaration and its data. Nothing rendered a screen, so we don't know whether the title key resolves.

The bigger problem is how the agents ran. To save time, we started them as subagents from an existing Claude Code session instead of in a clean CLI with an empty config directory. A subagent inherits the session's user and workspace instructions, and ours state the same rules the review checks: no any, the real test stack instead of a test dispatcher, no direct writes to event tables. Our results table has a column for rule breaks, and it was empty for all four runs. That says nothing about whether the package docs teach those rules. Only the columns for which docs an agent read and what it missed measure the package docs on their own.

We checked the transcripts for paths outside the project. None of the four agents touched our source tree or our config folder, and none used a skill or a web search.

What the eval found that wasn't documentation

The fresh scaffolds were broken in ways we never noticed, because we never start from a fresh scaffold. The prepare script had to patch some of this before an agent could start (version pins, the .gitignore, the Biome config, a random master key and admin password in .env), and none of those patches help with the tasks:

The scaffold wrote no .gitignore. Biome is configured to read it, so bun run lint failed in a brand new project.
Biome was pinned with a caret range that now resolves to a newer minor version. The new project was lint-red on its first run.
The generated production entry point seeded an admin user with a fixed password. Since 0.365.0 it requires DEMO_ADMIN_PASSWORD (at least 12 characters) in the env schema, and the boot stops before seeding when it is missing.

Upgrading the framework doesn't rewrite your bin/main.ts. If you scaffolded an app with 0.364.0 or earlier and booted it in production mode, change that admin's password and add the env variable by hand.

One of our own examples taught the wrong thing. The counter:active query in the custom-handlers sample loaded a page with list and filtered it in JavaScript. That only ever sees the first page, and it's the example an agent would copy when it needs to count. It now filters in the query and asks for totalCount.

While writing the error pages we also found that every docsUrl led to a 404. The framework built /errors/<code>, and the docs site served /en/errors/<code>/. The site now redirects, and a check fails the docs build when an error code has no page.

Next step

The run the eval was designed for is still open: one agent per project, started in the project directory with an empty Claude Code config directory, so nothing from our setup leaks in. The README in eval/agent-docs/ describes it, and that run is what tells us whether the rules in AGENTS.md reach an agent that has only the package.

The tasks, the hidden acceptance tests and the runner are in the public repo under eval/agent-docs/.

How do you grade an eval like this? We count "had to read compiled code" as a docs gap even when the task passes. We're not sure that's the right line, or whether it should fail the task.

Turning on MFA took one line. Making it true took three routes and a test kit

Marc — Thu, 15 Oct 2026 13:38:50 +0000

We are preparing Veridom, a property management app built on Kumiko, for a closed launch. One of the launch requirements was simple to state: every admin signs in with a second factor. In Kumiko that is a config value on the MFA feature:

return { issuer: BRAND_NAME, setupTokenSecret, challengeTokenSecret, requiredPolicy: "admins" };

The policy has three values: optional, admins and all. With admins, a password login by an admin who has no factor gets a setup step instead of a session, and an admin with a factor gets a challenge. The login handler enforces that.

The change was one line. Everything after it is what this post is about.

89 red tests, one message

The first effect was that the whole end-to-end suite went red. 89 Playwright specs, all failing with 401 on their first API call.

Every spec seeds a tenant and signs in its admin through a helper from our test kit. The helper posted the password, got back { mfaRequired: true, challengeToken } instead of a session cookie, and returned as if everything was fine. The 401 came one request later, from code that had nothing to do with login.

The fix for the suite was to teach the kit MFA. The more useful fix was to make the helper fail where the problem is:

if (reply.mfaRequired === true || reply.mfaSetupRequired === true) {
  throw new Error(
    `loginViaApi(${email}): the account needs MFA but no mfaTotpSecret was given; ` +
      `use seedTenant({ mfa: "totp" }) or addUser(roles, { mfa: "totp" })`,
  );
}

A test helper that swallows an unexpected state turns one clear failure into 89 confusing ones.

Teaching the kit TOTP

The kit now enrolls a real TOTP factor through the real endpoints. Under a policy that blocks the login, that is the pre-auth enrollment flow. Without one, it is the normal enrollment from a signed-in session. seedTenant() does this on its own when the admin's login answers with a setup step, so an app that switches on the policy does not have to touch every spec. Users created later can opt in with addUser(roles, { mfa: "totp" }), which matters when a spec creates a member and promotes them to admin afterwards.

One detail cost some thought. The server burns an accepted code for its whole validity window of one step either side, which is correct: a code seen once should not log anyone in twice. A suite that signs the same user in three times within 30 seconds then needs the next counter, and only codes up to one step ahead are valid. So the kit keeps the last counter per secret and, when it runs out of valid counters, waits until the clock catches up. That wait is tied to the TOTP window. It is not a test timeout, and we do not raise timeouts to make tests pass. In practice the full suite of 89 specs ran in 55 seconds with MFA on.

The policy had three doors around it

With the suite green, a security review of the release went through the diff. It found nothing blocking in the app. It found two problems in the framework and pointed at a third.

Switching tenants. A user can belong to several tenants. Picture someone who is a plain member in tenant A and an admin in tenant B, with no second factor. They sign in to A, where the policy does not apply to them, and get a session. Then they call POST /auth/switch-tenant for B. That route minted a fresh session for B without asking the MFA question. An admin session without a second factor, from a stolen password.

Accepting an invitation. Completing an invite signup issued the first session directly. That included admin invitations. The account holder could have enrolled a factor themselves, so this was hard to exploit, but it meant the first admin session of every invited admin had no second factor.

Signing up. Self-service signup creates a new tenant and makes the person who signed up its admin. Confirming the signup issued that admin session directly, again without the gate. Veridom launches without self-registration, so this one did not block us, but it is the same hole.

All three had the same root. The policy was enforced in the login handler, and these routes mint sessions without going through it. A policy flag is only as strong as the set of routes that create sessions, and nobody had made a list of that set.

The agent's first fix was worse than the bug

I gave the fix to a coding agent with a clear description: put the same MFA check in front of the switch-tenant session.

There is a layering wrinkle. The route lives in the framework core, the MFA gate lives in a bundled feature that the core cannot import. The agent solved that by creating a write handler for the gate and dispatching it from the route. Its first version took tenantId and roles from the request payload and was open to every signed-in user. Since all write handlers are reachable through the public write endpoint, anyone could have asked for an MFA challenge for any tenant, with any roles they liked. The second version restricted the handler to signed-in users and took the userId from the payload, which made it possible to trigger the gate for other people.

An automated security check flagged both versions while the agent was still working. I stopped it each time with the same rule, written out explicitly:

only the system identity may call the handler, and the route dispatches it as the system user;
the payload carries userId and tenantId, nothing else;
the handler resolves the membership and the roles itself, on the server, and returns an error when there is no membership;
a test calls the handler directly as a normal user and expects a 403.

That is the version we kept. The useful lesson for me: "put the same check here" is not a spec for auth code. The agent did what it was asked and filled the gaps with the most convenient assumption, which in auth code is the client's word. The rules that matter (who can call this, which inputs are trusted) have to be in the task, and a reviewer has to check them in the diff.

Tests that fail when the gate is gone

Each new test for the first two routes was checked by removing the thing it protects. Taking the gate out of the switch-tenant route turned two tests red. Taking it out of the invite signup turned one red. Opening the gate handler to normal users again turned four red, including the direct call.

The new integration tests run over real HTTP against a real stack and cover: switching to an admin tenant without a factor (setup step, no session cookie), with a factor (challenge, then a session after verifying), switching to a tenant where the user is only a member (direct session), switching to a tenant without membership (error, no token), the invite signup for an admin and for a member, and the signup confirmation under the policy, including a direct call through the public write endpoint.

Making the list

Each fix closes one route. To catch the next route that mints a session, the framework now keeps the list we were missing. Every production route mints its session through one function, and a guard test lists each of those routes together with how the MFA gate applies to it. A new route that mints a session without an entry fails the build, and so does a token signed anywhere else.

All of this shipped in Kumiko 0.345.0. Since 0.347.0, signing up under the policy also keeps the page the user wanted to land on after they enroll their factor. If you run Kumiko with an MFA policy, upgrade to at least 0.345.0.

If you run a multi-tenant app with per-tenant roles: does your MFA policy hold when a user switches tenants? We found out ours did not when a security review read the release that turned the policy on.

We moved OCR out of the app so egress could be zero

Marc — Wed, 30 Sep 2026 08:15:00 +0000

Our document ingest path used to run OCR inside the app pods. The library wanted glibc. The app images were Alpine. Missing language packs got pulled from github.com at parse time. That last part is the one that kept me up: a PDF upload could trigger an outbound fetch from a pod that otherwise had no business talking to the public internet.

So we moved parsing into its own service, baked the traineddata into the image, and closed egress entirely. Including DNS.

Why the app was the wrong place

Three separate failures stacked in one process:

The native liteparse binary didn't load on Alpine. You discover that in the target environment, not in a glibc laptop build.
Tessdata that wasn't already on disk downloaded at runtime. OCR "just worked" until the network policy, the rate limit, or the broken mirror made it not work.
The parse library accepted path-shaped options (tessdataPath, imageOutputDir, ocrServerUrl). A caller that could set those could point the parser at another filesystem or another host.

None of those are model problems. They're packaging and trust-boundary problems that show up the first time you treat document upload as untrusted input, which it is.

One HTTP service, three caller knobs

The service is a small Bun HTTP app: multipart upload in, pages of text out. Callers may pass only:

z.strictObject({
  ocrEnabled: z.boolean().optional(),
  ocrLanguage: z.string().regex(/^[a-z]{3}(\+[a-z]{3})*$/).optional(),
  maxPages: z.number().int().positive().optional(),
});

strictObject rejects every other field by name. tessdataPath is not an option; it comes from LITEPARSE_TESSDATA_PATH and nowhere else. Languages are checked against the .traineddata files present at startup. Ask for fra when only eng and deu are baked in and you get 400, not a download.

The image pins deu and eng from tesseract-ocr/tessdata_best at a fixed commit SHA with sha256 verification. Build time is when we fetch. Runtime is when we refuse to.

Auth is a shared bearer token, sixteen characters minimum. Health checks stay unauthenticated so Kubernetes can probe without holding the secret. There is no /metrics scrape path, because nothing scrapes it yet and an open metrics port would be another thing to reason about.

Closed egress is the point

In the cluster the service sits in its own namespace. The NetworkPolicy sets egress: [] — no outbound traffic, DNS included. Ingress is an allowlist of caller namespaces from stack config. Empty list means deny-all until the first consumer is declared.

That shape only works because the container never needs the network. Puppeteer, sitting next to it in our stack, cannot do the same: it renders user HTML with external subresources. OCR can, once the models live in the image.

App pods call http://liteparse-service…:3000 over the cluster network with the shared token. They never embed the native binary. Alpine stays Alpine. The parse blast radius is one dedicated Deployment with a memory tier and no way out.

What we'd do again

Pin traineddata in the image and fail closed on unknown languages. Whitelist request options so a multipart field cannot redirect filesystem or network paths. Put the NetworkPolicy next to the Deployment in the same PR, with ingress empty by default so a forgotten consumer config doesn't open the service to the whole cluster.

If your OCR still downloads language packs on first use, you don't have an offline parser. You have a deferred curl with a PDF as the trigger.

Undocumented handlers don't exist to the agent

Marc — Sun, 27 Sep 2026 06:10:00 +0000

We wired an in-app AI agent into a product that already had a full write/query surface. Auth was fine. The agent had permission to call writes. Half the handlers it needed simply never showed up in its tool list.

The agent said it couldn't find them. The handlers existed. What was missing was a description string.

What the agent can see

Our agent doesn't get raw TypeScript. It gets a manifest built from every feature the app mounts: handlers, screens, entities. Each entry needs a short description, written at the place the handler is defined. Without that string, the entry is dropped. The agent never learns the name exists.

That surprised a few people, including me the first time. We treated descriptions as documentation polish. For the agent they are the gate. An endpoint without an OpenAPI description might still be callable if you know the path. An agent-facing handler without a description isn't callable at all, because the model never sees a tool to choose.

Twenty-one handlers across our AI pipeline and prompt store features were in that state. The code ran for humans in the UI. The agent walked past them.

You also can't patch this later at the composition site. The description has to live next to the handler definition, so every app that mounts the feature inherits it. We learned that the hard way: people tried to "add docs" in the app and the gap tests kept failing, because the packaged feature still shipped blank.

Always allow is a permission, not a mood

Visibility is half of it. The other half is what happens after the agent picks a tool.

Writes default to risk mid. Queries default to low. The UI can offer "always allow" for those. Two of our writes are different:

{
  name: "delete-golden",
  description:
    "Permanently deletes a golden fixture, so it can no longer be used for dry-runs. This is a hard delete, not an appended revision, and cannot be undone.",
  agent: { risk: "high" },
  // ...
}

{
  name: "edit",
  description:
    "Saves new content for a prompt template by appending a revision that becomes the active prompt immediately… The owning AI feature uses this content verbatim as its system prompt…",
  agent: { risk: "high" },
  // ...
}

delete-golden is a hard delete. Prompt edit is free text that becomes another feature's system prompt on the next run. Both are marked high. The permission loop refuses always for those handlers outright. You can still approve a single call. You cannot teach the agent that this shape of write is permanently fine.

Most of the neighbouring writes append a restorable revision (set-policy, rollback, prompt revert). Those stay at the default. Restorable side effects and irreversible ones share a form in the code; they don't share a permission ceiling.

Phone apps already do this split. Camera access can be "always". Wipe storage can't. We needed the same ceiling for an agent that can press buttons faster than a human reviews them.

Make the gap fail a test

Once you decide "no description means invisible", silence becomes the failure mode. The agent looks underpowered. Nobody opens a ticket that says "missing string on handler 14".

So each feature that should be agent-visible gets a gap test:

test("createAiPipelineFeature([]) exposes 0 doc gaps", () => {
  const gaps = findAgentDocGaps([createAiPipelineFeature([])]);
  expect(gaps).toEqual([]);
});

test("delete-golden is high risk, set-policy is mid risk", () => {
  const feature = createAiPipelineFeature([]);
  expect(resolveAgentExposure(feature.writeHandlers["delete-golden"], "write").risk).toBe("high");
  expect(resolveAgentExposure(feature.writeHandlers["set-policy"], "write").risk).toBe("mid");
});

The first test is the smoke alarm for missing descriptions. The second pins the two risk decisions we actually care about, so a refactor can't quietly demote a hard delete back to "always allow"-eligible.

We also run the same gap lint from a CLI against a whole app config. Shipping a new consumer feature without descriptions turns red before anyone asks the agent to do a demo.

What we tell ourselves now

If a human can click it and an agent should be able to call it, write the description where the handler is defined. One sentence that says what changes and what stays reversible is enough; our pipeline descriptions are longer because the agent needs the follow-up query names (revisions before activate).

If the write can't be undone, or it rewrites another feature's brain, mark it high. Don't rely on the operator to remember which "always allow" clicks were a bad idea.

If you only remember one check: ask the agent to list what it can do, then compare that list to your write handlers. The missing names are usually missing strings, not missing models.

I wrote earlier about schemas beating prose for tool outputs. This is the other side of the same habit. There the description was advice and the schema was the contract. Here the description is the contract for whether the tool exists at all, and risk is the contract for whether "always" is even legal.

A green boot does not prove your key manager is being used

Marc — Fri, 25 Sep 2026 08:11:00 +0000

Five production apps, three encryption keys each, all of them sitting in a Kubernetes secret as base64 plaintext. The plan was to move them into Scaleway's Key Manager so the cluster only ever holds ciphertext and the pod unwraps it at boot. Straightforward work, one afternoon, maybe two.

It took three days, and every single failure mode was the same shape: something booted fine and was quietly wrong.

The slot that resolves twice

Our framework has a env schema per app. A field can be marked as a key manager slot:

KUMIKO_SECRETS_MASTER_KEY_V1: base64Key32
  .describe("AES-256 master key (KEK) for tenant-secrets encryption.")
  .meta({ kumiko: { kms: true } }),

At boot, the runtime walks those slots, finds the _CIPHERTEXT twin for each one, calls the key manager, and hands the app an env with the plaintext filled in. One round trip, everything resolved, done.

Except two of our five apps also build their own crypto provider before the framework's boot path runs, because they need it to construct a context object the framework then receives. So the same slot gets resolved twice, from two different env objects, in two different places. Our boot log made this visible only because both passes log with a prefix:

[publicstatus] KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par
[runProdApp]   KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par

I had been reading only the second line for a week. The first one is the one that would have crashed.

Plaintext next to ciphertext wins, silently

The first version of the Pulumi code set both values, on the theory that having the plaintext around was a safe fallback during the migration. It is the opposite of safe. The resolver checks for a plaintext value first, and if it finds one it never calls the key manager at all. You get a pod that boots, serves traffic, passes every health check, and has never once talked to the KMS you just spent two days wiring up.

So the deployment code is either/or, with a comment that says why:

// Either/or, never both: a plaintext slot left beside its ciphertext
// wins silently, so the pod would boot green while still bypassing the
// Key Manager.
...(secretsMasterKeyCiphertext
  ? { KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: secretsMasterKeyCiphertext }
  : { KUMIKO_SECRETS_MASTER_KEY_V1: masterKey.base64 }),

If you are doing this migration, make the two states mutually exclusive in code before you touch any environment. A fallback that can win without telling you is worse than no fallback.

The schema ate the key before the resolver saw it

The second pass resolves against the parsed env, and Zod drops unknown keys. The _CIPHERTEXT twin is generated by the framework for the composed schema, but each app also hand builds a typed schema on top. If the twin is not declared there, the parsed env simply does not contain it, and the resolver gets an env where the ciphertext does not exist and the plaintext is empty. It throws, the pod crashloops, and the error message talks about a missing key rather than a stripped one.

The fix is one line per app, and it needs a comment, because the next person will read it as redundant with the framework's generated twin and delete it:

KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: z.string().min(1).optional()
  .describe("Key-Manager ciphertext, wrapped by the same key as the KEK. Must stay in this schema: bin/main.ts hands the PARSED env to resolvePlatformKeks, which never sees an undeclared key."),

I actually deleted it myself, mid migration, after convincing myself it was dead code. It was not.

Zod 4 throws on extend for refined schemas

When a slot arrives as ciphertext only, the plaintext field has to be relaxed to optional before parsing, otherwise the required check rejects exactly the deployment the feature exists to enable. The obvious implementation is schema.extend({ [name]: field.optional() }).

That works for four of our apps. The fifth one has an object wide superRefine for a cross field rule, and Zod 4 throws on .extend() for any schema carrying a refinement. It failed at boot, in production, on a Friday. The version that works uses safeExtend:

return Object.keys(relaxed).length === 0 ? schema : schema.safeExtend(relaxed);

A framework helper that manipulates a user supplied schema has to be tested against a refined schema. That is the case that behaves differently, and it is never the case in your fixtures.

The app that did not mount the feature but needed the key

One app never mounts the tenant secrets feature, so its schema had no reason to know about the master key slot. It does use multi factor auth, and the MFA code envelope encrypts TOTP secrets with the same provider, pulling the key straight out of process.env.

Name based greps did not find it. The provider resolves its keys by regex over the environment:

const match = /^KUMIKO_SECRETS_MASTER_KEY_V(\d+)$/.exec(name);

So no consumer ever types the key name, and a grep for the name finds nothing in the one app that most needed the change. What found it was grepping for the provider's constructor instead. If you are auditing who consumes a secret, grep for the thing that reads it, not for the name of the thing being read.

Verifying that the key did not change

The whole migration is only safe if the wrapped value decrypts back to the exact same bytes. Existing rows in the database were encrypted with that key, and a key that is merely valid will decrypt nothing.

The check that proves this is narrow. Encrypt the live value, then decrypt the fresh ciphertext and compare the raw plaintext field from the API response against the source string. Do not compare your own encode of the decode, which passes at any nesting depth and proves only that your codec is symmetric. We had already been burned by exactly that once, which is why the wrap script aborts unless the raw fields match:

if [ "$roundtrip" != "$plaintext" ]; then
  echo "!! $app: decrypt of the fresh ciphertext does NOT match the source value" >&2
  exit 1
fi

A monitor that reported green on an unchecked commit

Small one, but it nearly cost me a bad merge. I had a poll loop watching four pull requests, filtering the GitHub check rollup for anything failing and anything pending. No failures and nothing pending reads as green.

Right after a force push the rollup is momentarily empty, so for about thirty seconds a branch with zero checks run against it satisfies both conditions. My monitor announced green on a commit that CI had not looked at yet. The rewrite uses the exit code of gh pr checks instead, where 0 is passed and 8 is pending, so an empty rollup cannot masquerade as success.

Whenever you derive a positive state from the absence of negatives, ask what an empty input looks like.

What actually went wrong on rollout day

Two things, neither of them about keys.

Four apps rolled at once and hit remaining connection slots are reserved for roles with the SUPERUSER attribute, because the old pods still held their Postgres connections while the new ones started. One app restarted and came back. Worth staging your rollouts if your connection ceiling is anywhere near your replica count times your pool size.

And the rollback path was not what I assumed. The Pulumi code reads the ciphertext with config.require(...), so removing the config value does not revert anything, it makes the next pulumi up throw before it can do any work. The real rollback is deploying the previous revision of the infrastructure code. That is a thing to work out before you need it rather than during.

Where it landed

All five apps now log source=key-manager for all three key slots, in both resolution passes, and the plaintext slot is gone from every Kubernetes secret. The first app went out alone with a targeted apply, the other four followed once its boot log confirmed the path.

The part I would repeat is the log line that names its own pass. Almost every wrong turn above was visible in that log before it was visible anywhere else, and one of them was only visible there.

Your AI eval is green because it never called the model

Marc — Tue, 22 Sep 2026 06:57:00 +0000

We run an eval suite on every PR. Seven fixtures, four metrics, deterministic, no API key, no network, costs nothing. It was green for weeks.

Then a code review found that two of the fixtures contained responses the live tool schema would have rejected outright. One had access nested inside a definition wrapper when the real pattern shape has it at the top level. Another declared a field as "json", which isn't a valid field type in our framework at all; the real one is "jsonb".

These are the canned responses the mock provider hands back. They were never checked against the schema the live model has to satisfy, because nothing in a mock run ever talks to a schema validator. So the suite was green on outputs that could not physically occur in production.

Every "test your LLM in CI" setup makes this trade, and it's worth stating plainly: a mocked eval tests your pipeline, never your model. Once you accept that, the mock gets more useful, because you stop asking it for something it can't give.

One runner, two providers

The entire trick is that the runner doesn't know which mode it's in. It takes a provider as a dependency and applies the same metric pipeline either way:

export type RunEvalOptions = {
  readonly provider: LLMProvider;
  readonly fixtures: readonly EvalFixture[];
  readonly metrics?: Readonly<Record<string, MetricFn>>;
  readonly mode: "mock" | "live";
  readonly generatedAt?: string;
};

mode is metadata. It lands in the report so you can tell a mock baseline from a live run later, and it changes no behavior. Mode selection lives in the script entry point, which is the only place that knows whether --live was passed.

For mock runs, the fixtures script themselves:

export async function runMockEval(options: RunMockEvalOptions): Promise<EvalReport> {
  const provider = createMockProvider();
  // Script every fixture's mockResponse in fixture-order. Runner
  // calls provider.chat() once per fixture, also in fixture-order,
  // so FIFO matches.
  for (const fixture of options.fixtures) {
    provider.script(fixture.mockResponse);
  }
  return runEval({ ...options, provider, mode: "mock" });
}

Each fixture carries its own mockResponse next to its expected outcome, so there's no separate mock-fixture directory to keep in sync with the real one. The provider is a FIFO queue and the runner is sequential, so the ordering holds without any matching logic.

Sequential is deliberate, by the way. Anthropic rate-limits, the eval set is seven fixtures, and parallelism would buy a few seconds in exchange for 429s in live mode.

Baseline diffs in review

A test suite that says "still green" after a prompt change is nearly worthless for this kind of work. Prompt and schema changes rarely break things outright, they shift them, and what you want to see is which fixture moved and by how much.

So the mock report goes into a checked-in JSON baseline, and a test compares a fresh mock run against it:

expect(live.summary.totalFixtures).toBe(baseline.summary.totalFixtures);
expect(live.summary.passing).toBe(baseline.summary.passing);
expect(live.summary.failing).toBe(baseline.summary.failing);
expect(live.summary.meanScore).toBeCloseTo(baseline.summary.meanScore, 6);

Plus per-fixture: same id set, same pass state, same aggregate score, same per-metric pass state.

Edit a fixture, a metric, the system prompt, or a tool definition, and this test goes red until you regenerate the baseline. The regenerated baseline is a file in your diff, so the reviewer sees parse-error flipping from true to false on one fixture, in the same PR that changed the tool schema. Quality changes become visible artifacts of review instead of something you notice three weeks later in production.

It also means "regenerate the baseline" is a normal, expected step, which is the part people resist. If regenerating feels like cheating, the baseline is doing its job: you have to look at the diff and decide the change was intended.

Two ways we got this wrong

The first was comparing the reason strings. Every metric result carries a human-readable message alongside its pass state and score. Comparing those in the drift test is tempting and wrong. The text drifts whenever someone improves an error message, and then the drift test fails for a reason that has nothing to do with model quality. After the third spurious failure people stop reading the diff, which kills the only thing the test is for. We compare structural data only: pass state, score, per-metric pass state. If a message needs to be load-bearing, it gets a dedicated metric test.

The second was letting the script pick the baseline filename. Live runs and mock runs write to different files, for the obvious reason that a live run must never clobber the deterministic baseline. That selection originally lived in a side-effectful script body, untested, where a swapped ternary would quietly destroy the mock baseline on the next live run. It's now one function with three tests, the third of which exists purely to state the invariant:

export function baselineFileName(isLive: boolean): string {
  return isLive ? "l2-eval-live-baseline.json" : "l2-eval-baseline.json";
}

test("the two modes never resolve to the same file", () => {
  expect(baselineFileName(true)).not.toBe(baselineFileName(false));
});

A test that asserts two constants differ looks silly right up until someone refactors the ternary.

Pass gates, score tracks

Every metric returns both a boolean and a 0..1 score, and the split matters more than it looks.

Pass is what gates CI. Score is what shows you a metric moving before it flips. Coverage of generated pattern kinds is a good example: four of five expected kinds found is 0.8, not zero, and not a pass.

const score = found / expected.length;
const pass = missing.length === 0;

The parse metric does the same, degrading instead of going binary. Clean parse is 1.0, N parse errors is max(0, 1 - N / 10), and a parser that throws is 0. So a change that takes a fixture from seven parse errors to two shows up as real movement even though it failed both times. Binary metrics hide exactly the progress you need to see while you're iterating on a prompt.

That metric parses the emitted source with the framework's real parser against an in-memory SourceFile. It's slower than a regex and it catches the case that matters: the model emits TypeScript that looks fine and our parser rejects, because it used factory style instead of the canonical form, or dropped the schema version header.

What the mock can't do

It cannot tell you the model got worse.

Everything above validates prompt assembly, tool definitions, metric logic, and report aggregation. All of that is code, all of it breaks in ordinary code ways, and testing it for free on every PR is worth doing. But the canned responses are ground truth by fiat. If the model starts emitting different shapes tomorrow, every one of these tests still passes.

That's what the live mode is for, and it's manual on purpose. A live run costs about $0.18 for the full set, which is nothing; the reason it isn't in CI is flakiness and rate limits, not money. It writes to its own baseline file and you fire it when you've changed something that could plausibly move model behavior. In the schema-tightening post the before-and-after numbers all came from live runs of this harness, for about $0.18 a round.

One detail that made live mode usable: a provider exception doesn't kill the run. The failing fixture gets a synthetic provider-error metric and the run continues, so a 429 on fixture three costs you one fixture instead of the whole report.

And the fixtures themselves need review like any other test data. Ours drifted because nobody was checking canned responses against the schema constraint that live mode enforces. That failure mode is intrinsic to mocking an LLM. You can only catch it by reading the fixtures, or by running live often enough that the gap shows up.

The short version

A mocked eval tests your pipeline. It cannot test your model, and it will happily stay green on responses the real API would reject.
Keep the runner mode-agnostic and pass the provider in. Both modes then share one metric pipeline, which is the thing you actually want identical.
Put the value in the checked-in baseline diff, so quality changes show up in review as a file someone has to read.
Compare structure, never human-readable messages, or the drift test becomes noise and people stop reading it.
Report pass and score separately: one gates, the other shows movement before it flips.
Review your fixtures against the constraints live mode enforces. Nothing else will.

The model obeys your schema, not your description

Marc — Fri, 18 Sep 2026 22:40:08 +0000

Two models. Same prompt, same tool description, same request. One of them returned this:

{ "kind": "entity", "entityName": "todo", "definition": { "fields": { "title": "text" } } }

The other returned this:

{ "kind": "entity", "name": "todo", "fields": { "title": "text" } }

The second one is wrong, and our downstream patcher rejected it with a 422 that told nobody anything useful. What took me a while to work out was why the first model got it right, because the answer turned out to have nothing to do with being smarter.

This was May 2026, Opus 4.7 and Sonnet 4.6 at the time. The story generalizes to whatever pair of models you're holding today.

The setup

We have a tool called apply_patches. An LLM reads a user request plus a source file and emits a list of structural change operations: add this entity, replace that handler, remove that metric. Each operation carries a pattern, the canonical object form of the thing being changed.

The tool schema for that pattern parameter was, in effect:

{ "type": "object", "properties": { "kind": { "type": "string" } }, "required": ["kind"] }

kind is a string and everything else is whatever. The actual shape lived in the tool description, a paragraph of prose with examples, the way most people write tool definitions.

The big model complied anyway. The smaller one didn't. Both had read the same description.

Why the big model complied

It had seen the shape before.

entityName and definition.fields are our field names, from our framework. To a model with training exposure to that shape, "emit an entity pattern" retrieves a memory. To a model without it, "emit an entity pattern" is a guess from the description text, and if you're guessing what an entity looks like, { name, fields } is a better guess than the truth. It's what everyone else's API would call those things.

So this is an exposure gap rather than a capability gap, which matters because you can't fix an exposure gap by paying for a bigger model. It will show up for any model on any shape that isn't in its training data, which is to say on your proprietary shapes, indefinitely. The less your schema looks like the rest of the internet, the harder your description has to work. And descriptions are not what the model is validated against.

The tool schema is. So we moved the contract into it.

Tight on the common kinds, loose on the tail

We have around twenty pattern kinds. Nine of them account for roughly 85% of everything the model emits. The other dozen (relation, workspace, secret, claimKey, systemScope and friends) show up rarely.

Writing strict schemas for all twenty would have been a week of work and a permanent maintenance tax, so we didn't. Each common kind became a discriminated oneOf branch with a real required list:

{
  title: "EntityPattern",
  properties: {
    kind: { const: "entity" },
    entityName: { type: "string" },
    definition: { type: "object" },
  },
  required: ["kind", "entityName", "definition"],
}

The long tail got one fallback branch that requires nothing but kind:

{
  title: "OtherPattern",
  properties: {
    kind: {
      type: "string",
      not: {
        enum: [
          "entity", "requires", "toggleable", "nav", "writeHandler",
          "queryHandler", "hook", "notification", "metric",
        ],
      },
    },
  },
  required: ["kind"],
}

Rare kinds still go through unvalidated at the schema layer, and the runtime patcher catches them. That split, tight on the discriminator values you see constantly and permissive on the ones you don't, is the part worth stealing. It costs an afternoon instead of a week and it targets the failures you actually get.

We did the same for the natural keys that replace and remove operations use, and pinned the per-operation requirements with allOf plus if/then, so the model can't hand us a replace with nothing to replace:

allOf: [
  { if: { properties: { op: { const: "replace" } } }, then: { required: ["id", "pattern"] } },
  { if: { properties: { op: { const: "add" } } },     then: { required: ["pattern"] } },
  { if: { properties: { op: { const: "remove" } } },  then: { required: ["id"] } },
]

Both Anthropic and OpenAI honor oneOf and allOf/if/then in tool input schemas. Most people skip them because the flat version works well enough on whichever model they tested with.

Did it work?

The evidence is thinner than I'd like, and it points the right way.

We ran three fixtures live, twice, for about $0.18 total. Before the change: two passed, one failed. The failure was the rename-entity case, emitting { kind: "entity", name, fields }. After: three passed, and the fixture that had been failing emitted { kind: "entity", entityName: "todo", definition: { fields: ... } }, byte for byte the shape the big model had been producing all along.

Three fixtures is not a benchmark. What convinced me was which failure disappeared and what replaced it. The smaller model stopped inventing field names and started producing the canonical shape, on the exact case that had been failing.

The real payoff came later. Once the smaller model could reliably emit the structured shape, it became viable as the default. That's usually the whole business case for schema work: a tight schema is what makes the cheap model good enough.

Footgun 1: oneOf is strict XOR

Two weeks later, a code review caught something the tests hadn't.

For replace and remove operations we have a parallel set of variants describing just the natural key. One of them is a singleton fallback: a kind and nothing else. And { kind: "entity" } matched both the entity branch and the fallback branch.

oneOf means exactly one. Two matches is a violation. Anthropic tolerated it; OpenAI's strict mode rejects the schema outright. Same schema, one provider silently fine, the other refusing to run.

The fix is the not.enum you saw above, where the fallback explicitly excludes every kind that has its own branch. Worth internalizing if you're building discriminated unions in JSON Schema: a fallback branch is not automatically disjoint from the specific ones. You have to make it disjoint by hand, and a test has to hold it that way, because the day someone adds a tenth specific branch and forgets the exclusion list is the day one of your providers starts 400ing.

Footgun 2: field order is a token budget problem

Different bug, same week. A generate_feature call came back with featureName, packageDescription, rationale, and no source. Source being the entire point of the call.

The model had emitted rationale first, written about 700 tokens of thoughtful design commentary, and hit maxTokens: 4000 before it got to the field that mattered. Nothing errored. We just got a well-argued explanation of a file that didn't exist.

Three fixes, in descending order of how much I trust them:

source.minLength: 100 and rationale.maxLength: 600, plus a blunt "BRIEF, 2-3 sentences" in the description. These are real constraints and they're what actually holds.
Raised maxTokens to 8000 for this one tool. The bounded tools stayed at 4000, since an unbounded budget everywhere just makes the truncation rarer and weirder.
Declared source first in the schema. Anthropic has been observed to emit arguments in declaration order. Observed, not documented. It costs nothing, it might help, and if a provider update changes it tomorrow nothing fails loudly. I left a comment in the source saying exactly that, and you should treat it the same way: a hint, not a contract.

Scores on the affected fixture went from 0.50 to 0.85, on both models, which is the tell that this was never a model-quality issue. The remaining 0.15 is genuine content quality: the model doesn't reach for one of our helpers when it should. That's a prompt and few-shot problem, and no amount of schema work will fix it. Knowing which of your failures are schema-shaped saves you from tightening things that were never loose.

The part nobody solves for you

These schemas are a hand-written mirror of our framework's real pattern types. Two definitions of the same shape, in two files, with nothing but discipline between them. When someone adds a required field to the real type, the schema doesn't know.

We pin what we can in a contract test: the number of variants, the required-field list per kind, the not.enum exclusion list. Drift fails a test instead of confusing a model in production six weeks later. That's a smoke alarm rather than a solution. If you generate your tool schemas from your actual types, you're ahead of us. If you're hand-writing them like we are, at least pin them.

What it costs

The obvious objection: you just made your prompt bigger, on every single call.

About 3KB bigger, in our case. With prompt caching that's one cache write at 1.25× input rate, roughly $0.0002 for the schema chunk, and every subsequent call in the cache window reads it at 10%. The schema sits in the stable prefix, which is exactly where caching is designed to put it.

Structured output used to carry a real tradeoff between schema size and bill. With caching it mostly doesn't. If token cost is what's keeping your tool definitions vague, go measure it. The number is probably smaller than the cost of one confusing 422 in production.

The short version

The description is advice. The schema is the contract. Models validate against one of those.
A model complying with your undocumented shape may just be recognizing it from training. That's not a capability you can rely on, and your next model isn't guaranteed to have it.
Tight schemas on the handful of kinds that dominate your traffic, one loose fallback for the tail. Don't schema the whole universe.
Make your fallback branch explicitly disjoint, or oneOf will bite you on the strictest provider you support.
Length constraints hold. Field order is a hint. Know which is which.
The reason to do any of this is usually that it makes the cheaper model good enough.

You asked how I know what the model did. Here are the answers, with numbers.

Marc — Thu, 06 Aug 2026 15:44:56 +0000

I wrote about our AI pipeline costs a while back. The comments were better than the post.

Valentin Monteiro made the point that cache alerts should be keyed per prompt version rather than on a global ratio, because a global number moves for boring reasons. Tae Kim pointed at how prompt caching fails silently, where a breakpoint on anything per-request gives you a miss on every call that logs exactly like a hit. Both need the same thing underneath, and I said I'd get cache metrics into our provenance records.

That turned into a bigger job than I expected. Here's what came out of it, what it costs, and the two parts I still haven't worked out.

Why store more than the output?

Because the output can't tell you whether something was always broken or broke last Tuesday, whether it's one record or ten thousand, or whether you changed something or the vendor did.

A total_amount column has the value. Nothing about where it came from.

Where do you record it?

At the provider, not in the feature.

I did it in the feature first. It worked fine. But then every new AI feature has to remember to do the same thing, and eventually one won't. So it moved down to where providers get built:

function withProvenance(provider, meta) {
  return async (request) => {
    const startedAt = performance.now()
    try {
      const response = await provider(request)
      await recordAiCall(buildAiCallPayload({ meta, request, response, startedAt }))
      return response
    } catch (error) {
      await recordAiCall(buildAiCallPayload({ meta, request, error, startedAt }))
      throw error
    }
  }
}

Nothing opts in because nothing gets asked.

One detail: this write goes outside the handler's transaction. If the business transaction rolls back, the call still happened and you still paid for it.

What's in the payload?

{
  "providerId": "anthropic",
  "handlerName": "invoice-extract",
  "requestedModel": "claude-sonnet-5",
  "respondedModel": "claude-sonnet-5-20260514",
  "promptVersion": "3f9a1c0e77b2",
  "inputHash": "9c4e1ab77f30d552",
  "latencyMs": 1180,
  "usage": {
    "inputTokens": 2140,
    "outputTokens": 318,
    "cacheCreationInputTokens": 0,
    "cacheReadInputTokens": 1890
  },
  "reportedCostUsd": 0.0042,
  "stopReason": "end_turn"
}

The two cache fields are there because of Valentin's comment. cacheReadInputTokens against inputTokens is your hit ratio, and now it's per call rather than a monthly average, so you can key an alert on it per prompt version like he suggested instead of watching one global number.

Requested and responded model are separate fields. Looks pedantic until a vendor routes you somewhere else and they don't match.

No prompt text, no output. Only hashes. The log lives forever and prompts are full of customer invoices.

Failures get a row too, with error kind and HTTP status. Most setups only log the successes.

How do you version a prompt?

I planned to put a version in the prompt file and bump it by hand. Then I didn't, because hand-maintained versions rot.

It hashes the stable, model-visible part of the request at runtime instead: system instructions, corpus, and the tool schemas. Twelve hex characters, and nobody bumps anything. Messages stay out on purpose, because if they were in it every call would get its own version and the thing would group nothing.

The "and the tool schemas" part is newer than this post. It used to hash the cacheable prefix only. Our extraction handler builds its tool from the caller's output schema, so you could change that schema, send the model a demonstrably different request, and promptVersion wouldn't move. inputHash saw it, but that one is unique per call, so it identifies without grouping. Which is the missed bump the runtime hash was supposed to make impossible. Found it writing this, wrote an issue, fixed it, merged it before the post went out. toolChoice went in at the same time, the field Tae flagged in the last thread ;)

What's left after that is key order, and it's a real one. JSON.stringify hashes the serialisation, so reordering properties in a tool schema flips the version without changing anything semantically.

The obvious move is to sort keys and hash a canonical form. We decided against it, and the reason is the interesting bit: the model sees the prompt as serialised text, and property order in a JSON schema affects the order an LLM generates fields in. Two differently sorted schemas are genuinely two different prompts. Sorting them into one bucket would rebuild exactly the missed bump we just removed, only invisibly. So the trade isn't false splits against churn. It's a visible false split against an invisible missed bump, and visible wins.

The other thing is on purpose. The hash is one-way, so you get 3f9a1c0e77b2 and no route back to the prompt. You can tell which calls are affected but not what changed in them. Reconstructing means git plus rehashing against the corpus from that day. The hand-bumped file version would have handled that with git blame. I didn't think about it until after I'd shipped the hash.

We have a prompt store with proper revision history sitting in the same codebase, connected to none of this. That's probably the answer and I haven't wired it up.

What does it cost?

Postgres 16, synthetic events, shape checked against real ones from the provider path:

513 bytes per payload at the median, 517 at p95
about 1,036 bytes per call with row overhead and the event store's three indexes
at 10k calls a day that's roughly 296 MB a month, 3.5 GB a year

Same per-call number at 10k, 100k and 1M rows. Storage isn't where the money goes.

Can you query it?

This is the part where measuring changed my mind.

"Every call with prompt version X" is what you run when something is wrong. An event store indexes tenant, aggregate type and time. Not the inside of a JSONB payload.

Scoped to one tenant and a time window: 162 ms over a million events. The index picks ~67,000 candidate rows and the payload filter narrows those to 8,490, so you pay for hauling 67,000 rows out of the heap.

Globally it's a seq scan, so you add an expression index on the payload field. At a million rows:

distinct prompt versions	no index	with index	plan
8	115 ms	281 ms	ignored it, seq scan
50	107 ms	61 ms	bitmap heap scan
500	107 ms	1.02 ms	bitmap heap scan

Index costs 6.9 MB in all three.

With eight versions one version is 12.5% of the table, so Postgres scans and is right to. The index just sits there.

The problem is that day one is when you benchmark this, see nothing, and decide the index isn't worth it. Six months in you've edited prompts a few hundred times, one version is 0.2% of the table, and that same index is a hundred times faster. Add it once you're past single digits.

Do I need event sourcing?

No. An append-only ai_calls table gets you nearly all of it. We already had the event log so it came out of that for free.

The bit worth copying either way is where you put the recording.

What would you change before this goes live?

Asking for real. It's built and merged but not out to a live tenant yet, so this is still a good moment to hear that it's wrong.

Which is also why there are no cost or latency numbers from real traffic in here. We don't have them yet. In two months I will.

If you've run something like this for a while: what's in your call log that isn't in mine? And where does this fall apart at a volume I haven't hit?

The benchmark SQL is in the repo if you'd rather measure your own database than trust mine. It's self-contained, so psql -v rows=1000000 -v prompt_versions=500 -f ai-call-provenance-benchmark.sql against a throwaway database reproduces the table above.

I build Kumiko, an event sourced framework for multi-tenant B2B systems in TypeScript.

Multi-Tenant AI Chat: From Hardcoded Config to BYOK in 4 Steps

Marc — Mon, 03 Aug 2026 08:35:50 +0000

Two tenants, two AI providers, two prompts. Sounds simple, and on day one it is. That's the trap. It stays simple right up until customer number two sends their first "quick question," and eighteen months later you're running a small distributed system to answer it. Here's the honest version of that slide, four stages, each one caused by a real human typing a real request into Slack.

Day 1: it just works (he says, foolishly) 😅

Tenant A wants OpenAI. Tenant B wants Claude. Both want their own system prompt. The obvious first version: one config object, one row per tenant. What could possibly go wrong. (Everything. Everything could go wrong. But not yet.)

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│  Request    │ →   │  TENANT_CONFIG   │ →   │  Provider   │
│  (tenantId) │     │  (hardcoded obj) │     │  SDK call   │
└─────────────┘     └──────────────────┘     └─────────────┘

const TENANT_AI_CONFIG = {
  tenantA: { provider: 'openai', model: 'gpt-5', prompt: 'You are terse and technical.' },
  tenantB: { provider: 'anthropic', model: 'claude-sonnet-5', prompt: 'You are friendly. Antworte auf Deutsch.' },
} as const

async function handleChat(tenantId: string, userMessage: string) {
  const config = TENANT_AI_CONFIG[tenantId]
  const client = config.provider === 'openai' ? openai : anthropic
  return client.chat(config.model, config.prompt, userMessage)
}

Ships in an afternoon. Two tenants, two rows, demo goes great, everyone claps 👏. Put this moment in a frame, it's the calmest the codebase will ever be.

Evolution 1: Tenant B wants their own twist

A week in (a week, we didn't even get a full sprint), tenant B messages: "can we change the prompt ourselves, without waiting for a deploy?" Fair ask, they know their users, we don't, and also nobody wants to be the on-call engineer who gets paged to edit a string literal. A hardcoded object can't answer that, it needs a rebuild to change a comma.

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│  Request    │ →   │  tenant_settings │ →   │  Provider   │
│  (tenantId) │     │  (DB row, admin  │     │  SDK call   │
│             │     │   editable)      │     │             │
└─────────────┘     └──────────────────┘     └─────────────┘

Config moves from a code constant to a table the tenant's own admin UI can write to.

async function getAiConfig(tenantId: string) {
  const row = await ctx.db.tenantSettings.findOne({ tenantId })
  return row.aiConfig // { provider, model, prompt }
}

Same shape as before, different source. handleChat doesn't change at all, it has no idea any of this happened, which is exactly the point of putting the lookup behind one function.

Evolution 2: "wait, who changed the prompt?" 🕵️

Self-service is great until it isn't: tenant B's prompt quietly changed last Tuesday, their bot started answering in pirate-speak for reasons nobody can reconstruct, support gets a ticket, and the honest answer is "we have no idea, the database doesn't remember either." A plain DB row just gets overwritten, the past has no representation, it's Ctrl+Z with no undo history.

┌──────────────┐     ┌────────────────┐     ┌──────────────────┐
│  Admin edits │ →   │  ConfigChanged │ →   │  current config  │
│  the prompt  │     │  event (who,   │     │  = fold(events)  │
│              │     │  when, diff)   │     │                  │
└──────────────┘     └────────────────┘     └──────────────────┘

The config becomes event-sourced instead of a mutable row: every change is an event, the current value is a projection over them.

async function updateAiConfig(tenantId: string, patch: Partial<AiConfig>, actor: string) {
  await ctx.emit('AiConfigChanged', { tenantId, patch, actor, at: ctx.now() })
}

async function getAiConfig(tenantId: string): Promise<AiConfig> {
  const events = await ctx.db.events.find({ tenantId, type: 'AiConfigChanged' })
  return events.reduce((cfg, e) => ({ ...cfg, ...e.patch }), DEFAULT_AI_CONFIG)
}

Now "who changed it and when" is a query, not a seance 🔮. handleChat still hasn't changed, it just calls getAiConfig, blissfully unaware it's now talking to an event log instead of a table.

Evolution 3: BYOK and a usage cap 💸

A bigger tenant shows up, the kind that gets its own Slack channel, with two demands: they want to use their own OpenAI key (cost control, their own rate limits, their own finance team breathing down their neck), and they want a hard cap on monthly spend so an over-caffeinated intern's script can't turn into a five-figure invoice.

┌──────────────┐     ┌───────────────────────┐     ┌──────────────┐
│  Request     │ →   │  config.apiKey?       │ →   │  usage < cap?│
│              │     │  (BYOK, encrypted)    │     │  → call      │
│              │     │  else our shared key  │     │  → else 429  │
└──────────────┘     └───────────────────────┘     └──────────────┘

async function callProvider(tenantId: string, userMessage: string) {
  const config = await getAiConfig(tenantId)
  const usage = await getMonthlyUsage(tenantId)
  if (config.usageCap && usage >= config.usageCap) {
    throw new UsageCapExceeded(tenantId)
  }

  const apiKey = config.byokApiKey ?? process.env.SHARED_API_KEY // BYOK overrides shared key
  const client = getClient(config.provider, apiKey)
  const reply = await client.chat(config.model, config.prompt, userMessage)

  await recordUsage(tenantId, reply.usage.totalTokens)
  return reply
}

Two additive fields on the same config, byokApiKey and usageCap, and one counter check before the call. No new architecture, no rewrite of the first three stages, no "sorry, we need a full quarter to redesign this."

The pattern behind the pattern 🧵

Every stage kept getAiConfig(tenantId) → { provider, model, prompt, ... } as the seam. Storage changed underneath it four times (constant, DB row, event-sourced projection, projection with encrypted secrets) and the call site never noticed, never cared, never even asked. That's the actual lesson: don't design the multi-tenant AI system upfront, design one seam that can absorb whatever the next Slack message throws at it.

Skipped on purpose: provider fallback, streaming, per-model cost tables. Add those when a tenant actually asks, same as everything above. If a tenant asks for a fifth provider before you've read this sentence, that's not a counterexample, that's Tuesday. 🙃