DEV Community: Marc

A green boot does not prove your key manager is being used

Marc — Sat, 03 Oct 2026 06:15:00 +0000

Five production apps, three encryption keys each, all stored as base64-encoded plaintext in Kubernetes Secrets. The plan was to move the values into Scaleway's Key Manager so Kubernetes would store only ciphertext and the pod would unwrap it at boot. Straightforward work, one afternoon, maybe two.

It took three days. Every failure mode had the same shape: something booted fine and was quietly wrong.

The slot that resolves twice

Our framework has an environment schema for each app. A field can be marked as a key-manager slot:

KUMIKO_SECRETS_MASTER_KEY_V1: base64Key32
  .describe("AES-256 master key (KEK) for tenant-secrets encryption.")
  .meta({ kumiko: { kms: true } }),

At boot, the runtime walks those slots, finds the _CIPHERTEXT twin for each one, calls the key manager, and hands the app an env with the plaintext filled in. One resolution pass, everything resolved, done.

Two of our five apps also build their own crypto provider before the framework's boot path runs. They need it to construct a context object that the framework receives later. So the same slot gets resolved twice, from two different env objects, in two different places. Our boot log made this visible only because both passes log with a prefix:

[publicstatus] KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par
[runProdApp]   KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par

I had been reading only the second line for a week. Before the fix, the first pass was the one that would have crashed.

Plaintext next to ciphertext wins, silently

The first version of the Pulumi code set both values, on the theory that having the plaintext around was a safe fallback during the migration. It is the opposite of safe. The resolver checks for a plaintext value first, and if it finds one it never calls the key manager at all. You get a pod that boots, serves traffic, passes every health check, and has never once talked to the KMS you just spent two days wiring up.

So the deployment code is either/or, with a comment that says why:

// Either/or, never both: a plaintext slot left beside its ciphertext
// wins silently, so the pod would boot green while still bypassing the
// Key Manager.
...(secretsMasterKeyCiphertext
  ? { KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: secretsMasterKeyCiphertext }
  : { KUMIKO_SECRETS_MASTER_KEY_V1: masterKey.base64 }),

If you are doing this migration, make the two states mutually exclusive in code before you touch any environment. A fallback that can win silently is worse than no fallback.

The schema ate the key before the resolver saw it

The second pass resolves against the parsed env, and Zod's default object parsing drops unknown keys. The _CIPHERTEXT twin is generated by the framework for the composed schema, but each app also hand builds a typed schema on top. If the twin is not declared there, the parsed env simply does not contain it, and the resolver gets an env where the ciphertext is missing and the plaintext is still absent. It throws, the pod crashloops, and the error message talks about a missing key rather than a stripped one.

The fix is one line per app, and it needs an explanation, because the next person will read it as redundant with the framework's generated twin and delete it:

KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: z.string().min(1).optional()
  .describe("Key-Manager ciphertext, wrapped by the same key as the KEK. Must stay in this schema: bin/main.ts hands the PARSED env to resolvePlatformKeks, which never sees an undeclared key."),

I actually deleted it myself, mid migration, after convincing myself it was dead code. It was not.

Zod 4 throws on extend for refined schemas

When a slot arrives as ciphertext only, the plaintext field has to be relaxed to optional before parsing, otherwise the required check rejects exactly the deployment the feature exists to enable. The obvious implementation is schema.extend({ [name]: field.optional() }).

That works for four of our apps. The fifth one has an object-wide superRefine for a cross-field rule, and Zod 4 throws on .extend() for any schema carrying a refinement. It failed at boot, in production, on a Friday. The version that works uses safeExtend:

return Object.keys(relaxed).length === 0 ? schema : schema.safeExtend(relaxed);

A framework helper that manipulates a user-supplied schema has to be tested against a refined schema. That is the case that behaves differently, and it is never the case in your fixtures.

The app that did not mount the feature but needed the key

One app never mounts the tenant secrets feature, so its schema had no reason to know about the master key slot. It does use multifactor authentication, and the MFA code envelope encrypts TOTP secrets with the same provider, pulling the key straight out of process.env.

Name-based greps did not find it. The provider resolves its keys by regex over the environment:

const match = /^KUMIKO_SECRETS_MASTER_KEY_V(\d+)$/.exec(name);

So no consumer ever types the key name, and a grep for the name finds nothing in the app that most needed the change. What found it was grepping for the provider's constructor instead. If you are auditing who consumes a secret, grep for the thing that reads it, not for the name of the thing being read.

Verifying that the key did not change

The migration is safe only if the wrapped value decrypts to the exact same key bytes. Existing rows in the database were encrypted with that key, and a key that is merely valid will decrypt nothing.

The check is narrow. Encrypt the live value, then decrypt the fresh ciphertext and compare the API's raw plaintext field with the source string, using the same canonical encoding. Do not compare your own encode of the decode; that proves only that your codec is symmetric. We had already been burned by exactly that once, which is why the wrap script aborts unless the raw fields match:

if [ "$roundtrip" != "$plaintext" ]; then
  echo "!! $app: decrypt of the fresh ciphertext does NOT match the source value" >&2
  exit 1
fi

A monitor that reported green on an unchecked commit

Small one, but it nearly cost me a bad merge. I had a poll loop watching four pull requests, filtering the GitHub check rollup for anything failing and anything pending. No failures and nothing pending reads as green.

Right after a force push, the rollup is momentarily empty, so for about thirty seconds a branch with zero checks run against it satisfies both conditions. My monitor announced green on a commit that CI had not looked at yet. The rewrite uses the exit code of gh pr checks instead, where 0 is passed and 8 is pending, so an empty rollup cannot masquerade as success.

Whenever you derive a positive state from the absence of negatives, ask what an empty input looks like.

What actually went wrong on rollout day

Two other things went wrong, both unrelated to keys.

Four apps rolled at once and hit remaining connection slots are reserved for roles with the SUPERUSER attribute, because the old pods still held their Postgres connections while the new ones started. One app restarted and came back. Worth staging your rollouts if your connection ceiling is anywhere near your replica count times your pool size.

The rollback path was also different from what I assumed. The Pulumi code reads the ciphertext with config.require(...), so removing the config value does not revert anything. It makes the next pulumi up throw before it can do any work. The real rollback is to deploy the previous revision of the infrastructure code. That is a thing to work out before you need it rather than during.

Where it landed

All five apps now log source=key-manager for all three key slots. The two apps with the extra boot path log it in both resolution passes, and the plaintext slot is gone from every Kubernetes secret. The first app went out alone with a targeted apply, the other four followed once its boot log confirmed the path.

The part I would repeat is the log line that names its own pass. Almost every wrong turn above showed up in that log before it showed up anywhere else, and one showed up only there.

We moved OCR out of the app so egress could be zero

Marc — Wed, 30 Sep 2026 08:15:00 +0000

Our document ingest path used to run OCR inside the app pods. The library wanted glibc. The app images were Alpine. Missing language packs got pulled from github.com at parse time. That last part is the one that kept me up: a PDF upload could trigger an outbound fetch from a pod that otherwise had no business talking to the public internet.

So we moved parsing into its own service, baked the traineddata into the image, and closed egress entirely. Including DNS.

Why the app was the wrong place

Three separate failures stacked in one process:

The native liteparse binary didn't load on Alpine. You discover that in the target environment, not in a glibc laptop build.
Tessdata that wasn't already on disk downloaded at runtime. OCR "just worked" until the network policy, the rate limit, or the broken mirror made it not work.
The parse library accepted path-shaped options (tessdataPath, imageOutputDir, ocrServerUrl). A caller that could set those could point the parser at another filesystem or another host.

None of those are model problems. They're packaging and trust-boundary problems that show up the first time you treat document upload as untrusted input, which it is.

One HTTP service, three caller knobs

The service is a small Bun HTTP app: multipart upload in, pages of text out. Callers may pass only:

z.strictObject({
  ocrEnabled: z.boolean().optional(),
  ocrLanguage: z.string().regex(/^[a-z]{3}(\+[a-z]{3})*$/).optional(),
  maxPages: z.number().int().positive().optional(),
});

strictObject rejects every other field by name. tessdataPath is not an option; it comes from LITEPARSE_TESSDATA_PATH and nowhere else. Languages are checked against the .traineddata files present at startup. Ask for fra when only eng and deu are baked in and you get 400, not a download.

The image pins deu and eng from tesseract-ocr/tessdata_best at a fixed commit SHA with sha256 verification. Build time is when we fetch. Runtime is when we refuse to.

Auth is a shared bearer token, sixteen characters minimum. Health checks stay unauthenticated so Kubernetes can probe without holding the secret. There is no /metrics scrape path, because nothing scrapes it yet and an open metrics port would be another thing to reason about.

Closed egress is the point

In the cluster the service sits in its own namespace. The NetworkPolicy sets egress: [] — no outbound traffic, DNS included. Ingress is an allowlist of caller namespaces from stack config. Empty list means deny-all until the first consumer is declared.

That shape only works because the container never needs the network. Puppeteer, sitting next to it in our stack, cannot do the same: it renders user HTML with external subresources. OCR can, once the models live in the image.

App pods call http://liteparse-service…:3000 over the cluster network with the shared token. They never embed the native binary. Alpine stays Alpine. The parse blast radius is one dedicated Deployment with a memory tier and no way out.

What we'd do again

Pin traineddata in the image and fail closed on unknown languages. Whitelist request options so a multipart field cannot redirect filesystem or network paths. Put the NetworkPolicy next to the Deployment in the same PR, with ingress empty by default so a forgotten consumer config doesn't open the service to the whole cluster.

If your OCR still downloads language packs on first use, you don't have an offline parser. You have a deferred curl with a PDF as the trigger.

Undocumented handlers don't exist to the agent

Marc — Sun, 27 Sep 2026 06:10:00 +0000

We wired an in-app AI agent into a product that already had a full write/query surface. Auth was fine. The agent had permission to call writes. Half the handlers it needed simply never showed up in its tool list.

The agent said it couldn't find them. The handlers existed. What was missing was a description string.

What the agent can see

Our agent doesn't get raw TypeScript. It gets a manifest built from every feature the app mounts: handlers, screens, entities. Each entry needs a short description, written at the place the handler is defined. Without that string, the entry is dropped. The agent never learns the name exists.

That surprised a few people, including me the first time. We treated descriptions as documentation polish. For the agent they are the gate. An endpoint without an OpenAPI description might still be callable if you know the path. An agent-facing handler without a description isn't callable at all, because the model never sees a tool to choose.

Twenty-one handlers across our AI pipeline and prompt store features were in that state. The code ran for humans in the UI. The agent walked past them.

You also can't patch this later at the composition site. The description has to live next to the handler definition, so every app that mounts the feature inherits it. We learned that the hard way: people tried to "add docs" in the app and the gap tests kept failing, because the packaged feature still shipped blank.

Always allow is a permission, not a mood

Visibility is half of it. The other half is what happens after the agent picks a tool.

Writes default to risk mid. Queries default to low. The UI can offer "always allow" for those. Two of our writes are different:

{
  name: "delete-golden",
  description:
    "Permanently deletes a golden fixture, so it can no longer be used for dry-runs. This is a hard delete, not an appended revision, and cannot be undone.",
  agent: { risk: "high" },
  // ...
}

{
  name: "edit",
  description:
    "Saves new content for a prompt template by appending a revision that becomes the active prompt immediately… The owning AI feature uses this content verbatim as its system prompt…",
  agent: { risk: "high" },
  // ...
}

delete-golden is a hard delete. Prompt edit is free text that becomes another feature's system prompt on the next run. Both are marked high. The permission loop refuses always for those handlers outright. You can still approve a single call. You cannot teach the agent that this shape of write is permanently fine.

Most of the neighbouring writes append a restorable revision (set-policy, rollback, prompt revert). Those stay at the default. Restorable side effects and irreversible ones share a form in the code; they don't share a permission ceiling.

Phone apps already do this split. Camera access can be "always". Wipe storage can't. We needed the same ceiling for an agent that can press buttons faster than a human reviews them.

Make the gap fail a test

Once you decide "no description means invisible", silence becomes the failure mode. The agent looks underpowered. Nobody opens a ticket that says "missing string on handler 14".

So each feature that should be agent-visible gets a gap test:

test("createAiPipelineFeature([]) exposes 0 doc gaps", () => {
  const gaps = findAgentDocGaps([createAiPipelineFeature([])]);
  expect(gaps).toEqual([]);
});

test("delete-golden is high risk, set-policy is mid risk", () => {
  const feature = createAiPipelineFeature([]);
  expect(resolveAgentExposure(feature.writeHandlers["delete-golden"], "write").risk).toBe("high");
  expect(resolveAgentExposure(feature.writeHandlers["set-policy"], "write").risk).toBe("mid");
});

The first test is the smoke alarm for missing descriptions. The second pins the two risk decisions we actually care about, so a refactor can't quietly demote a hard delete back to "always allow"-eligible.

We also run the same gap lint from a CLI against a whole app config. Shipping a new consumer feature without descriptions turns red before anyone asks the agent to do a demo.

What we tell ourselves now

If a human can click it and an agent should be able to call it, write the description where the handler is defined. One sentence that says what changes and what stays reversible is enough; our pipeline descriptions are longer because the agent needs the follow-up query names (revisions before activate).

If the write can't be undone, or it rewrites another feature's brain, mark it high. Don't rely on the operator to remember which "always allow" clicks were a bad idea.

If you only remember one check: ask the agent to list what it can do, then compare that list to your write handlers. The missing names are usually missing strings, not missing models.

I wrote earlier about schemas beating prose for tool outputs. This is the other side of the same habit. There the description was advice and the schema was the contract. Here the description is the contract for whether the tool exists at all, and risk is the contract for whether "always" is even legal.

A green boot does not prove your key manager is being used

Marc — Fri, 25 Sep 2026 08:11:00 +0000

Five production apps, three encryption keys each, all of them sitting in a Kubernetes secret as base64 plaintext. The plan was to move them into Scaleway's Key Manager so the cluster only ever holds ciphertext and the pod unwraps it at boot. Straightforward work, one afternoon, maybe two.

It took three days, and every single failure mode was the same shape: something booted fine and was quietly wrong.

The slot that resolves twice

Our framework has a env schema per app. A field can be marked as a key manager slot:

KUMIKO_SECRETS_MASTER_KEY_V1: base64Key32
  .describe("AES-256 master key (KEK) for tenant-secrets encryption.")
  .meta({ kumiko: { kms: true } }),

At boot, the runtime walks those slots, finds the _CIPHERTEXT twin for each one, calls the key manager, and hands the app an env with the plaintext filled in. One round trip, everything resolved, done.

Except two of our five apps also build their own crypto provider before the framework's boot path runs, because they need it to construct a context object the framework then receives. So the same slot gets resolved twice, from two different env objects, in two different places. Our boot log made this visible only because both passes log with a prefix:

[publicstatus] KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par
[runProdApp]   KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par

I had been reading only the second line for a week. The first one is the one that would have crashed.

Plaintext next to ciphertext wins, silently

So the deployment code is either/or, with a comment that says why:

// Either/or, never both: a plaintext slot left beside its ciphertext
// wins silently, so the pod would boot green while still bypassing the
// Key Manager.
...(secretsMasterKeyCiphertext
  ? { KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: secretsMasterKeyCiphertext }
  : { KUMIKO_SECRETS_MASTER_KEY_V1: masterKey.base64 }),

If you are doing this migration, make the two states mutually exclusive in code before you touch any environment. A fallback that can win without telling you is worse than no fallback.

The schema ate the key before the resolver saw it

The second pass resolves against the parsed env, and Zod drops unknown keys. The _CIPHERTEXT twin is generated by the framework for the composed schema, but each app also hand builds a typed schema on top. If the twin is not declared there, the parsed env simply does not contain it, and the resolver gets an env where the ciphertext does not exist and the plaintext is empty. It throws, the pod crashloops, and the error message talks about a missing key rather than a stripped one.

The fix is one line per app, and it needs a comment, because the next person will read it as redundant with the framework's generated twin and delete it:

KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: z.string().min(1).optional()
  .describe("Key-Manager ciphertext, wrapped by the same key as the KEK. Must stay in this schema: bin/main.ts hands the PARSED env to resolvePlatformKeks, which never sees an undeclared key."),

I actually deleted it myself, mid migration, after convincing myself it was dead code. It was not.

Zod 4 throws on extend for refined schemas

That works for four of our apps. The fifth one has an object wide superRefine for a cross field rule, and Zod 4 throws on .extend() for any schema carrying a refinement. It failed at boot, in production, on a Friday. The version that works uses safeExtend:

return Object.keys(relaxed).length === 0 ? schema : schema.safeExtend(relaxed);

A framework helper that manipulates a user supplied schema has to be tested against a refined schema. That is the case that behaves differently, and it is never the case in your fixtures.

The app that did not mount the feature but needed the key

One app never mounts the tenant secrets feature, so its schema had no reason to know about the master key slot. It does use multi factor auth, and the MFA code envelope encrypts TOTP secrets with the same provider, pulling the key straight out of process.env.

Name based greps did not find it. The provider resolves its keys by regex over the environment:

const match = /^KUMIKO_SECRETS_MASTER_KEY_V(\d+)$/.exec(name);

So no consumer ever types the key name, and a grep for the name finds nothing in the one app that most needed the change. What found it was grepping for the provider's constructor instead. If you are auditing who consumes a secret, grep for the thing that reads it, not for the name of the thing being read.

Verifying that the key did not change

The whole migration is only safe if the wrapped value decrypts back to the exact same bytes. Existing rows in the database were encrypted with that key, and a key that is merely valid will decrypt nothing.

The check that proves this is narrow. Encrypt the live value, then decrypt the fresh ciphertext and compare the raw plaintext field from the API response against the source string. Do not compare your own encode of the decode, which passes at any nesting depth and proves only that your codec is symmetric. We had already been burned by exactly that once, which is why the wrap script aborts unless the raw fields match:

if [ "$roundtrip" != "$plaintext" ]; then
  echo "!! $app: decrypt of the fresh ciphertext does NOT match the source value" >&2
  exit 1
fi

A monitor that reported green on an unchecked commit

Right after a force push the rollup is momentarily empty, so for about thirty seconds a branch with zero checks run against it satisfies both conditions. My monitor announced green on a commit that CI had not looked at yet. The rewrite uses the exit code of gh pr checks instead, where 0 is passed and 8 is pending, so an empty rollup cannot masquerade as success.

Whenever you derive a positive state from the absence of negatives, ask what an empty input looks like.

What actually went wrong on rollout day

Two things, neither of them about keys.

And the rollback path was not what I assumed. The Pulumi code reads the ciphertext with config.require(...), so removing the config value does not revert anything, it makes the next pulumi up throw before it can do any work. The real rollback is deploying the previous revision of the infrastructure code. That is a thing to work out before you need it rather than during.

Where it landed

All five apps now log source=key-manager for all three key slots, in both resolution passes, and the plaintext slot is gone from every Kubernetes secret. The first app went out alone with a targeted apply, the other four followed once its boot log confirmed the path.

The part I would repeat is the log line that names its own pass. Almost every wrong turn above was visible in that log before it was visible anywhere else, and one of them was only visible there.

Your AI eval is green because it never called the model

Marc — Tue, 22 Sep 2026 06:57:00 +0000

We run an eval suite on every PR. Seven fixtures, four metrics, deterministic, no API key, no network, costs nothing. It was green for weeks.

Then a code review found that two of the fixtures contained responses the live tool schema would have rejected outright. One had access nested inside a definition wrapper when the real pattern shape has it at the top level. Another declared a field as "json", which isn't a valid field type in our framework at all; the real one is "jsonb".

These are the canned responses the mock provider hands back. They were never checked against the schema the live model has to satisfy, because nothing in a mock run ever talks to a schema validator. So the suite was green on outputs that could not physically occur in production.

Every "test your LLM in CI" setup makes this trade, and it's worth stating plainly: a mocked eval tests your pipeline, never your model. Once you accept that, the mock gets more useful, because you stop asking it for something it can't give.

One runner, two providers

The entire trick is that the runner doesn't know which mode it's in. It takes a provider as a dependency and applies the same metric pipeline either way:

export type RunEvalOptions = {
  readonly provider: LLMProvider;
  readonly fixtures: readonly EvalFixture[];
  readonly metrics?: Readonly<Record<string, MetricFn>>;
  readonly mode: "mock" | "live";
  readonly generatedAt?: string;
};

mode is metadata. It lands in the report so you can tell a mock baseline from a live run later, and it changes no behavior. Mode selection lives in the script entry point, which is the only place that knows whether --live was passed.

For mock runs, the fixtures script themselves:

export async function runMockEval(options: RunMockEvalOptions): Promise<EvalReport> {
  const provider = createMockProvider();
  // Script every fixture's mockResponse in fixture-order. Runner
  // calls provider.chat() once per fixture, also in fixture-order,
  // so FIFO matches.
  for (const fixture of options.fixtures) {
    provider.script(fixture.mockResponse);
  }
  return runEval({ ...options, provider, mode: "mock" });
}

Each fixture carries its own mockResponse next to its expected outcome, so there's no separate mock-fixture directory to keep in sync with the real one. The provider is a FIFO queue and the runner is sequential, so the ordering holds without any matching logic.

Sequential is deliberate, by the way. Anthropic rate-limits, the eval set is seven fixtures, and parallelism would buy a few seconds in exchange for 429s in live mode.

Baseline diffs in review

A test suite that says "still green" after a prompt change is nearly worthless for this kind of work. Prompt and schema changes rarely break things outright, they shift them, and what you want to see is which fixture moved and by how much.

So the mock report goes into a checked-in JSON baseline, and a test compares a fresh mock run against it:

expect(live.summary.totalFixtures).toBe(baseline.summary.totalFixtures);
expect(live.summary.passing).toBe(baseline.summary.passing);
expect(live.summary.failing).toBe(baseline.summary.failing);
expect(live.summary.meanScore).toBeCloseTo(baseline.summary.meanScore, 6);

Plus per-fixture: same id set, same pass state, same aggregate score, same per-metric pass state.

Edit a fixture, a metric, the system prompt, or a tool definition, and this test goes red until you regenerate the baseline. The regenerated baseline is a file in your diff, so the reviewer sees parse-error flipping from true to false on one fixture, in the same PR that changed the tool schema. Quality changes become visible artifacts of review instead of something you notice three weeks later in production.

It also means "regenerate the baseline" is a normal, expected step, which is the part people resist. If regenerating feels like cheating, the baseline is doing its job: you have to look at the diff and decide the change was intended.

Two ways we got this wrong

The first was comparing the reason strings. Every metric result carries a human-readable message alongside its pass state and score. Comparing those in the drift test is tempting and wrong. The text drifts whenever someone improves an error message, and then the drift test fails for a reason that has nothing to do with model quality. After the third spurious failure people stop reading the diff, which kills the only thing the test is for. We compare structural data only: pass state, score, per-metric pass state. If a message needs to be load-bearing, it gets a dedicated metric test.

The second was letting the script pick the baseline filename. Live runs and mock runs write to different files, for the obvious reason that a live run must never clobber the deterministic baseline. That selection originally lived in a side-effectful script body, untested, where a swapped ternary would quietly destroy the mock baseline on the next live run. It's now one function with three tests, the third of which exists purely to state the invariant:

export function baselineFileName(isLive: boolean): string {
  return isLive ? "l2-eval-live-baseline.json" : "l2-eval-baseline.json";
}

test("the two modes never resolve to the same file", () => {
  expect(baselineFileName(true)).not.toBe(baselineFileName(false));
});

A test that asserts two constants differ looks silly right up until someone refactors the ternary.

Pass gates, score tracks

Every metric returns both a boolean and a 0..1 score, and the split matters more than it looks.

Pass is what gates CI. Score is what shows you a metric moving before it flips. Coverage of generated pattern kinds is a good example: four of five expected kinds found is 0.8, not zero, and not a pass.

const score = found / expected.length;
const pass = missing.length === 0;

The parse metric does the same, degrading instead of going binary. Clean parse is 1.0, N parse errors is max(0, 1 - N / 10), and a parser that throws is 0. So a change that takes a fixture from seven parse errors to two shows up as real movement even though it failed both times. Binary metrics hide exactly the progress you need to see while you're iterating on a prompt.

That metric parses the emitted source with the framework's real parser against an in-memory SourceFile. It's slower than a regex and it catches the case that matters: the model emits TypeScript that looks fine and our parser rejects, because it used factory style instead of the canonical form, or dropped the schema version header.

What the mock can't do

It cannot tell you the model got worse.

Everything above validates prompt assembly, tool definitions, metric logic, and report aggregation. All of that is code, all of it breaks in ordinary code ways, and testing it for free on every PR is worth doing. But the canned responses are ground truth by fiat. If the model starts emitting different shapes tomorrow, every one of these tests still passes.

That's what the live mode is for, and it's manual on purpose. A live run costs about $0.18 for the full set, which is nothing; the reason it isn't in CI is flakiness and rate limits, not money. It writes to its own baseline file and you fire it when you've changed something that could plausibly move model behavior. In the schema-tightening post the before-and-after numbers all came from live runs of this harness, for about $0.18 a round.

One detail that made live mode usable: a provider exception doesn't kill the run. The failing fixture gets a synthetic provider-error metric and the run continues, so a 429 on fixture three costs you one fixture instead of the whole report.

And the fixtures themselves need review like any other test data. Ours drifted because nobody was checking canned responses against the schema constraint that live mode enforces. That failure mode is intrinsic to mocking an LLM. You can only catch it by reading the fixtures, or by running live often enough that the gap shows up.

The short version

A mocked eval tests your pipeline. It cannot test your model, and it will happily stay green on responses the real API would reject.
Keep the runner mode-agnostic and pass the provider in. Both modes then share one metric pipeline, which is the thing you actually want identical.
Put the value in the checked-in baseline diff, so quality changes show up in review as a file someone has to read.
Compare structure, never human-readable messages, or the drift test becomes noise and people stop reading it.
Report pass and score separately: one gates, the other shows movement before it flips.
Review your fixtures against the constraints live mode enforces. Nothing else will.

The model obeys your schema, not your description

Marc — Fri, 18 Sep 2026 22:40:08 +0000

Two models. Same prompt, same tool description, same request. One of them returned this:

{ "kind": "entity", "entityName": "todo", "definition": { "fields": { "title": "text" } } }

The other returned this:

{ "kind": "entity", "name": "todo", "fields": { "title": "text" } }

The second one is wrong, and our downstream patcher rejected it with a 422 that told nobody anything useful. What took me a while to work out was why the first model got it right, because the answer turned out to have nothing to do with being smarter.

This was May 2026, Opus 4.7 and Sonnet 4.6 at the time. The story generalizes to whatever pair of models you're holding today.

The setup

We have a tool called apply_patches. An LLM reads a user request plus a source file and emits a list of structural change operations: add this entity, replace that handler, remove that metric. Each operation carries a pattern, the canonical object form of the thing being changed.

The tool schema for that pattern parameter was, in effect:

{ "type": "object", "properties": { "kind": { "type": "string" } }, "required": ["kind"] }

kind is a string and everything else is whatever. The actual shape lived in the tool description, a paragraph of prose with examples, the way most people write tool definitions.

The big model complied anyway. The smaller one didn't. Both had read the same description.

Why the big model complied

It had seen the shape before.

entityName and definition.fields are our field names, from our framework. To a model with training exposure to that shape, "emit an entity pattern" retrieves a memory. To a model without it, "emit an entity pattern" is a guess from the description text, and if you're guessing what an entity looks like, { name, fields } is a better guess than the truth. It's what everyone else's API would call those things.

So this is an exposure gap rather than a capability gap, which matters because you can't fix an exposure gap by paying for a bigger model. It will show up for any model on any shape that isn't in its training data, which is to say on your proprietary shapes, indefinitely. The less your schema looks like the rest of the internet, the harder your description has to work. And descriptions are not what the model is validated against.

The tool schema is. So we moved the contract into it.

Tight on the common kinds, loose on the tail

We have around twenty pattern kinds. Nine of them account for roughly 85% of everything the model emits. The other dozen (relation, workspace, secret, claimKey, systemScope and friends) show up rarely.

Writing strict schemas for all twenty would have been a week of work and a permanent maintenance tax, so we didn't. Each common kind became a discriminated oneOf branch with a real required list:

{
  title: "EntityPattern",
  properties: {
    kind: { const: "entity" },
    entityName: { type: "string" },
    definition: { type: "object" },
  },
  required: ["kind", "entityName", "definition"],
}

The long tail got one fallback branch that requires nothing but kind:

{
  title: "OtherPattern",
  properties: {
    kind: {
      type: "string",
      not: {
        enum: [
          "entity", "requires", "toggleable", "nav", "writeHandler",
          "queryHandler", "hook", "notification", "metric",
        ],
      },
    },
  },
  required: ["kind"],
}

Rare kinds still go through unvalidated at the schema layer, and the runtime patcher catches them. That split, tight on the discriminator values you see constantly and permissive on the ones you don't, is the part worth stealing. It costs an afternoon instead of a week and it targets the failures you actually get.

We did the same for the natural keys that replace and remove operations use, and pinned the per-operation requirements with allOf plus if/then, so the model can't hand us a replace with nothing to replace:

allOf: [
  { if: { properties: { op: { const: "replace" } } }, then: { required: ["id", "pattern"] } },
  { if: { properties: { op: { const: "add" } } },     then: { required: ["pattern"] } },
  { if: { properties: { op: { const: "remove" } } },  then: { required: ["id"] } },
]

Both Anthropic and OpenAI honor oneOf and allOf/if/then in tool input schemas. Most people skip them because the flat version works well enough on whichever model they tested with.

Did it work?

The evidence is thinner than I'd like, and it points the right way.

We ran three fixtures live, twice, for about $0.18 total. Before the change: two passed, one failed. The failure was the rename-entity case, emitting { kind: "entity", name, fields }. After: three passed, and the fixture that had been failing emitted { kind: "entity", entityName: "todo", definition: { fields: ... } }, byte for byte the shape the big model had been producing all along.

Three fixtures is not a benchmark. What convinced me was which failure disappeared and what replaced it. The smaller model stopped inventing field names and started producing the canonical shape, on the exact case that had been failing.

The real payoff came later. Once the smaller model could reliably emit the structured shape, it became viable as the default. That's usually the whole business case for schema work: a tight schema is what makes the cheap model good enough.

Footgun 1: oneOf is strict XOR

Two weeks later, a code review caught something the tests hadn't.

For replace and remove operations we have a parallel set of variants describing just the natural key. One of them is a singleton fallback: a kind and nothing else. And { kind: "entity" } matched both the entity branch and the fallback branch.

oneOf means exactly one. Two matches is a violation. Anthropic tolerated it; OpenAI's strict mode rejects the schema outright. Same schema, one provider silently fine, the other refusing to run.

The fix is the not.enum you saw above, where the fallback explicitly excludes every kind that has its own branch. Worth internalizing if you're building discriminated unions in JSON Schema: a fallback branch is not automatically disjoint from the specific ones. You have to make it disjoint by hand, and a test has to hold it that way, because the day someone adds a tenth specific branch and forgets the exclusion list is the day one of your providers starts 400ing.

Footgun 2: field order is a token budget problem

Different bug, same week. A generate_feature call came back with featureName, packageDescription, rationale, and no source. Source being the entire point of the call.

The model had emitted rationale first, written about 700 tokens of thoughtful design commentary, and hit maxTokens: 4000 before it got to the field that mattered. Nothing errored. We just got a well-argued explanation of a file that didn't exist.

Three fixes, in descending order of how much I trust them:

source.minLength: 100 and rationale.maxLength: 600, plus a blunt "BRIEF, 2-3 sentences" in the description. These are real constraints and they're what actually holds.
Raised maxTokens to 8000 for this one tool. The bounded tools stayed at 4000, since an unbounded budget everywhere just makes the truncation rarer and weirder.
Declared source first in the schema. Anthropic has been observed to emit arguments in declaration order. Observed, not documented. It costs nothing, it might help, and if a provider update changes it tomorrow nothing fails loudly. I left a comment in the source saying exactly that, and you should treat it the same way: a hint, not a contract.

Scores on the affected fixture went from 0.50 to 0.85, on both models, which is the tell that this was never a model-quality issue. The remaining 0.15 is genuine content quality: the model doesn't reach for one of our helpers when it should. That's a prompt and few-shot problem, and no amount of schema work will fix it. Knowing which of your failures are schema-shaped saves you from tightening things that were never loose.

The part nobody solves for you

These schemas are a hand-written mirror of our framework's real pattern types. Two definitions of the same shape, in two files, with nothing but discipline between them. When someone adds a required field to the real type, the schema doesn't know.

We pin what we can in a contract test: the number of variants, the required-field list per kind, the not.enum exclusion list. Drift fails a test instead of confusing a model in production six weeks later. That's a smoke alarm rather than a solution. If you generate your tool schemas from your actual types, you're ahead of us. If you're hand-writing them like we are, at least pin them.

What it costs

The obvious objection: you just made your prompt bigger, on every single call.

About 3KB bigger, in our case. With prompt caching that's one cache write at 1.25× input rate, roughly $0.0002 for the schema chunk, and every subsequent call in the cache window reads it at 10%. The schema sits in the stable prefix, which is exactly where caching is designed to put it.

Structured output used to carry a real tradeoff between schema size and bill. With caching it mostly doesn't. If token cost is what's keeping your tool definitions vague, go measure it. The number is probably smaller than the cost of one confusing 422 in production.

The short version

The description is advice. The schema is the contract. Models validate against one of those.
A model complying with your undocumented shape may just be recognizing it from training. That's not a capability you can rely on, and your next model isn't guaranteed to have it.
Tight schemas on the handful of kinds that dominate your traffic, one loose fallback for the tail. Don't schema the whole universe.
Make your fallback branch explicitly disjoint, or oneOf will bite you on the strictest provider you support.
Length constraints hold. Field order is a hint. Know which is which.
The reason to do any of this is usually that it makes the cheaper model good enough.

You asked how I know what the model did. Here are the answers, with numbers.

Marc — Thu, 06 Aug 2026 15:44:56 +0000

I wrote about our AI pipeline costs a while back. The comments were better than the post.

Valentin Monteiro made the point that cache alerts should be keyed per prompt version rather than on a global ratio, because a global number moves for boring reasons. Tae Kim pointed at how prompt caching fails silently, where a breakpoint on anything per-request gives you a miss on every call that logs exactly like a hit. Both need the same thing underneath, and I said I'd get cache metrics into our provenance records.

That turned into a bigger job than I expected. Here's what came out of it, what it costs, and the two parts I still haven't worked out.

Why store more than the output?

Because the output can't tell you whether something was always broken or broke last Tuesday, whether it's one record or ten thousand, or whether you changed something or the vendor did.

A total_amount column has the value. Nothing about where it came from.

Where do you record it?

At the provider, not in the feature.

I did it in the feature first. It worked fine. But then every new AI feature has to remember to do the same thing, and eventually one won't. So it moved down to where providers get built:

function withProvenance(provider, meta) {
  return async (request) => {
    const startedAt = performance.now()
    try {
      const response = await provider(request)
      await recordAiCall(buildAiCallPayload({ meta, request, response, startedAt }))
      return response
    } catch (error) {
      await recordAiCall(buildAiCallPayload({ meta, request, error, startedAt }))
      throw error
    }
  }
}

Nothing opts in because nothing gets asked.

One detail: this write goes outside the handler's transaction. If the business transaction rolls back, the call still happened and you still paid for it.

What's in the payload?

{
  "providerId": "anthropic",
  "handlerName": "invoice-extract",
  "requestedModel": "claude-sonnet-5",
  "respondedModel": "claude-sonnet-5-20260514",
  "promptVersion": "3f9a1c0e77b2",
  "inputHash": "9c4e1ab77f30d552",
  "latencyMs": 1180,
  "usage": {
    "inputTokens": 2140,
    "outputTokens": 318,
    "cacheCreationInputTokens": 0,
    "cacheReadInputTokens": 1890
  },
  "reportedCostUsd": 0.0042,
  "stopReason": "end_turn"
}

The two cache fields are there because of Valentin's comment. cacheReadInputTokens against inputTokens is your hit ratio, and now it's per call rather than a monthly average, so you can key an alert on it per prompt version like he suggested instead of watching one global number.

Requested and responded model are separate fields. Looks pedantic until a vendor routes you somewhere else and they don't match.

No prompt text, no output. Only hashes. The log lives forever and prompts are full of customer invoices.

Failures get a row too, with error kind and HTTP status. Most setups only log the successes.

How do you version a prompt?

I planned to put a version in the prompt file and bump it by hand. Then I didn't, because hand-maintained versions rot.

It hashes the stable, model-visible part of the request at runtime instead: system instructions, corpus, and the tool schemas. Twelve hex characters, and nobody bumps anything. Messages stay out on purpose, because if they were in it every call would get its own version and the thing would group nothing.

The "and the tool schemas" part is newer than this post. It used to hash the cacheable prefix only. Our extraction handler builds its tool from the caller's output schema, so you could change that schema, send the model a demonstrably different request, and promptVersion wouldn't move. inputHash saw it, but that one is unique per call, so it identifies without grouping. Which is the missed bump the runtime hash was supposed to make impossible. Found it writing this, wrote an issue, fixed it, merged it before the post went out. toolChoice went in at the same time, the field Tae flagged in the last thread ;)

What's left after that is key order, and it's a real one. JSON.stringify hashes the serialisation, so reordering properties in a tool schema flips the version without changing anything semantically.

The obvious move is to sort keys and hash a canonical form. We decided against it, and the reason is the interesting bit: the model sees the prompt as serialised text, and property order in a JSON schema affects the order an LLM generates fields in. Two differently sorted schemas are genuinely two different prompts. Sorting them into one bucket would rebuild exactly the missed bump we just removed, only invisibly. So the trade isn't false splits against churn. It's a visible false split against an invisible missed bump, and visible wins.

The other thing is on purpose. The hash is one-way, so you get 3f9a1c0e77b2 and no route back to the prompt. You can tell which calls are affected but not what changed in them. Reconstructing means git plus rehashing against the corpus from that day. The hand-bumped file version would have handled that with git blame. I didn't think about it until after I'd shipped the hash.

We have a prompt store with proper revision history sitting in the same codebase, connected to none of this. That's probably the answer and I haven't wired it up.

What does it cost?

Postgres 16, synthetic events, shape checked against real ones from the provider path:

513 bytes per payload at the median, 517 at p95
about 1,036 bytes per call with row overhead and the event store's three indexes
at 10k calls a day that's roughly 296 MB a month, 3.5 GB a year

Same per-call number at 10k, 100k and 1M rows. Storage isn't where the money goes.

Can you query it?

This is the part where measuring changed my mind.

"Every call with prompt version X" is what you run when something is wrong. An event store indexes tenant, aggregate type and time. Not the inside of a JSONB payload.

Scoped to one tenant and a time window: 162 ms over a million events. The index picks ~67,000 candidate rows and the payload filter narrows those to 8,490, so you pay for hauling 67,000 rows out of the heap.

Globally it's a seq scan, so you add an expression index on the payload field. At a million rows:

distinct prompt versions	no index	with index	plan
8	115 ms	281 ms	ignored it, seq scan
50	107 ms	61 ms	bitmap heap scan
500	107 ms	1.02 ms	bitmap heap scan

Index costs 6.9 MB in all three.

With eight versions one version is 12.5% of the table, so Postgres scans and is right to. The index just sits there.

The problem is that day one is when you benchmark this, see nothing, and decide the index isn't worth it. Six months in you've edited prompts a few hundred times, one version is 0.2% of the table, and that same index is a hundred times faster. Add it once you're past single digits.

Do I need event sourcing?

No. An append-only ai_calls table gets you nearly all of it. We already had the event log so it came out of that for free.

The bit worth copying either way is where you put the recording.

What would you change before this goes live?

Asking for real. It's built and merged but not out to a live tenant yet, so this is still a good moment to hear that it's wrong.

Which is also why there are no cost or latency numbers from real traffic in here. We don't have them yet. In two months I will.

If you've run something like this for a while: what's in your call log that isn't in mine? And where does this fall apart at a volume I haven't hit?

The benchmark SQL is in the repo if you'd rather measure your own database than trust mine. It's self-contained, so psql -v rows=1000000 -v prompt_versions=500 -f ai-call-provenance-benchmark.sql against a throwaway database reproduces the table above.

I build Kumiko, an event sourced framework for multi-tenant B2B systems in TypeScript.

Multi-Tenant AI Chat: From Hardcoded Config to BYOK in 4 Steps

Marc — Mon, 03 Aug 2026 08:35:50 +0000

Two tenants, two AI providers, two prompts. Sounds simple, and on day one it is. That's the trap. It stays simple right up until customer number two sends their first "quick question," and eighteen months later you're running a small distributed system to answer it. Here's the honest version of that slide, four stages, each one caused by a real human typing a real request into Slack.

Day 1: it just works (he says, foolishly) 😅

Tenant A wants OpenAI. Tenant B wants Claude. Both want their own system prompt. The obvious first version: one config object, one row per tenant. What could possibly go wrong. (Everything. Everything could go wrong. But not yet.)

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│  Request    │ →   │  TENANT_CONFIG   │ →   │  Provider   │
│  (tenantId) │     │  (hardcoded obj) │     │  SDK call   │
└─────────────┘     └──────────────────┘     └─────────────┘

const TENANT_AI_CONFIG = {
  tenantA: { provider: 'openai', model: 'gpt-5', prompt: 'You are terse and technical.' },
  tenantB: { provider: 'anthropic', model: 'claude-sonnet-5', prompt: 'You are friendly. Antworte auf Deutsch.' },
} as const

async function handleChat(tenantId: string, userMessage: string) {
  const config = TENANT_AI_CONFIG[tenantId]
  const client = config.provider === 'openai' ? openai : anthropic
  return client.chat(config.model, config.prompt, userMessage)
}

Ships in an afternoon. Two tenants, two rows, demo goes great, everyone claps 👏. Put this moment in a frame, it's the calmest the codebase will ever be.

Evolution 1: Tenant B wants their own twist

A week in (a week, we didn't even get a full sprint), tenant B messages: "can we change the prompt ourselves, without waiting for a deploy?" Fair ask, they know their users, we don't, and also nobody wants to be the on-call engineer who gets paged to edit a string literal. A hardcoded object can't answer that, it needs a rebuild to change a comma.

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│  Request    │ →   │  tenant_settings │ →   │  Provider   │
│  (tenantId) │     │  (DB row, admin  │     │  SDK call   │
│             │     │   editable)      │     │             │
└─────────────┘     └──────────────────┘     └─────────────┘

Config moves from a code constant to a table the tenant's own admin UI can write to.

async function getAiConfig(tenantId: string) {
  const row = await ctx.db.tenantSettings.findOne({ tenantId })
  return row.aiConfig // { provider, model, prompt }
}

Same shape as before, different source. handleChat doesn't change at all, it has no idea any of this happened, which is exactly the point of putting the lookup behind one function.

Evolution 2: "wait, who changed the prompt?" 🕵️

Self-service is great until it isn't: tenant B's prompt quietly changed last Tuesday, their bot started answering in pirate-speak for reasons nobody can reconstruct, support gets a ticket, and the honest answer is "we have no idea, the database doesn't remember either." A plain DB row just gets overwritten, the past has no representation, it's Ctrl+Z with no undo history.

┌──────────────┐     ┌────────────────┐     ┌──────────────────┐
│  Admin edits │ →   │  ConfigChanged │ →   │  current config  │
│  the prompt  │     │  event (who,   │     │  = fold(events)  │
│              │     │  when, diff)   │     │                  │
└──────────────┘     └────────────────┘     └──────────────────┘

The config becomes event-sourced instead of a mutable row: every change is an event, the current value is a projection over them.

async function updateAiConfig(tenantId: string, patch: Partial<AiConfig>, actor: string) {
  await ctx.emit('AiConfigChanged', { tenantId, patch, actor, at: ctx.now() })
}

async function getAiConfig(tenantId: string): Promise<AiConfig> {
  const events = await ctx.db.events.find({ tenantId, type: 'AiConfigChanged' })
  return events.reduce((cfg, e) => ({ ...cfg, ...e.patch }), DEFAULT_AI_CONFIG)
}

Now "who changed it and when" is a query, not a seance 🔮. handleChat still hasn't changed, it just calls getAiConfig, blissfully unaware it's now talking to an event log instead of a table.

Evolution 3: BYOK and a usage cap 💸

A bigger tenant shows up, the kind that gets its own Slack channel, with two demands: they want to use their own OpenAI key (cost control, their own rate limits, their own finance team breathing down their neck), and they want a hard cap on monthly spend so an over-caffeinated intern's script can't turn into a five-figure invoice.

┌──────────────┐     ┌───────────────────────┐     ┌──────────────┐
│  Request     │ →   │  config.apiKey?       │ →   │  usage < cap?│
│              │     │  (BYOK, encrypted)    │     │  → call      │
│              │     │  else our shared key  │     │  → else 429  │
└──────────────┘     └───────────────────────┘     └──────────────┘

async function callProvider(tenantId: string, userMessage: string) {
  const config = await getAiConfig(tenantId)
  const usage = await getMonthlyUsage(tenantId)
  if (config.usageCap && usage >= config.usageCap) {
    throw new UsageCapExceeded(tenantId)
  }

  const apiKey = config.byokApiKey ?? process.env.SHARED_API_KEY // BYOK overrides shared key
  const client = getClient(config.provider, apiKey)
  const reply = await client.chat(config.model, config.prompt, userMessage)

  await recordUsage(tenantId, reply.usage.totalTokens)
  return reply
}

Two additive fields on the same config, byokApiKey and usageCap, and one counter check before the call. No new architecture, no rewrite of the first three stages, no "sorry, we need a full quarter to redesign this."

The pattern behind the pattern 🧵

Every stage kept getAiConfig(tenantId) → { provider, model, prompt, ... } as the seam. Storage changed underneath it four times (constant, DB row, event-sourced projection, projection with encrypted secrets) and the call site never noticed, never cared, never even asked. That's the actual lesson: don't design the multi-tenant AI system upfront, design one seam that can absorb whatever the next Slack message throws at it.

Skipped on purpose: provider fallback, streaming, per-model cost tables. Add those when a tenant actually asks, same as everything above. If a tenant asks for a fifth provider before you've read this sentence, that's not a counterexample, that's Tuesday. 🙃

We Cut Our AI Pipeline Costs 25% Without Losing Accuracy (and the fix wasn't a cheaper model)

Marc — Sat, 01 Aug 2026 18:46:54 +0000

Our AI pipeline runs three step kinds (ai.generate, ai.extract, ai.classify), each independently resolving its own provider, model, and prompt revision at run time. The default model is Sonnet, not Opus — and for a while that felt like a compromise, because Opus was the expensive-but-reliable option and Sonnet needed babysitting to hit the same pass rate.

The fix that closed the gap wasn't a smarter prompt. It was tool_choice plus a tightened output schema, forcing the model to commit to an answer shape instead of spending tokens hedging its way there. That alone got Sonnet to pass-parity with Opus's prior output at roughly a quarter of the cost, in our eval. A second, separate lever: Anthropic's own recommended effort: "xhigh" for agentic tasks produced roughly twice the thinking tokens of "high" for the same pass rate — thinking tokens bill like output tokens, so that's a straight 2x for zero accuracy gain, and it only matters when Opus is used via an explicit override (Sonnet doesn't get adaptive thinking at all). A third, independent lever: max_tokens defaulted to 16000 out of caution; dropping it to 4000 (still comfortably above what any real step needed) cut effective output cost again, because Anthropic bills against the cap as an upper bound in some failure paths, not only against what the model actually emits.

None of these three touched the prompt content or the provider. All three came from looking at per-step token usage, which only exists because every step writes an immutable provenance record on completion: prompt revision, provider, model, token usage. Same idea as event sourcing, applied to LLM calls instead of domain writes. You don't trust "the pipeline probably used prompt v3, on whatever model was configured," you have a row that says so — and you can audit six months of "which prompt touched this tenant" without ever loading the actual LLM payloads (also the DSGVO-friendly shape: provenance rows carry no user content, only call metadata).

The other real trap, found the hard way: adaptive thinking and forced tool-use don't mix. Anthropic 400s with "Thinking may not be enabled when tool_choice forces tool use" the moment both are set, which sounds obvious in hindsight but not when you're setting both because both individually sound like "make the model try harder." The fix has to be automatic — detect a forced tool_choice and disable adaptive thinking before the request goes out — because leaving it to every caller to remember means every eval run using toolChoice: { type: "any" } against an Opus override breaks the same way, all at once.

Separately: prompt caching only pays off if the cache_control breakpoint sits on the last static block, with tools and system prompt as one cacheable prefix. Past roughly 16k tokens without it, requests start hitting the SDK's HTTP timeout before the response streams back far enough to matter. Put the breakpoint on something that changes per request and you get a silent cache-miss that looks identical to a hit in the logs while you keep paying full price.

Full writeup with the pipeline architecture (with diagrams), the provider-resolution code, and the provenance shape: docs.kumiko.rocks/en/guides/ai-pipeline-provenance.

We Shipped a Mortgage Calculator Bug Where Every Test Was Green and the Answer Was Still Wrong

Marc — Sat, 01 Aug 2026 11:48:07 +0000

A review pass on our credit calculator flagged something that should have been impossible: a
finding tagged HIGH, on a panel with a full test suite, where every individual test passed and
the answer was still wrong. Not "wrong in an edge case" wrong. Wrong for every single user who had
ever configured a Sondertilgung (a lump-sum extra payment on their mortgage) and then looked at
the "what if I invest the difference instead" comparison next to it.

The bug: the comparison panel always ran the loan projection with an empty additionals list. The
headline result above it, the number the user actually configured and trusted, included their
real Sondertilgungen. So if you had a 20k lump-sum payment scheduled for month 6, the headline
said "your loan is basically gone by year 8" and the panel right next to it, presented as
directly comparable, was quietly running a different loan. Not a rounding difference. A different
payoff date, a different remaining balance, a different verdict on whether investing instead
would have won.

Every test was green because every test checked one function against one input. Nothing checked
that the two numbers on the same screen were talking about the same loan.

Why "pay down debt faster vs. invest the difference" is rigged by default

That specific bug is a symptom of a more general trap. Almost every calculator online that lets
you compare "pay down debt faster" against "invest the difference instead" gets the comparison
itself wrong, independent of any coding bug — because it quietly changes the monthly budget
between the two scenarios and presents the result as a fair fight.

Say you have a 300k loan at 3.5%, paying 3% annual amortization. Someone suggests: "drop to 1%
amortization, invest the rest." The naive calculator compares:

Scenario A: 3% amortization, no investing, ~1200/month total outflow
Scenario B: 1% amortization, invest a fixed 200/month, ~950/month total outflow

Of course B looks better over ten years. You're comparing spending 1200/month against spending
950/month and marveling that the cheaper option built more wealth. It didn't win on insight. It
won because you fed it a smaller number.

The fix is a structural invariant, not a footnote:

credit installment (this month) + ETF contribution (this month) = fixed monthly budget

Every month, for both scenarios, and once a loan is paid off in one scenario, its full budget
rolls into the ETF contribution from that point on. You don't get to just stop spending because
the mortgage happened to end early in the model. Once that holds, "total interest paid" stops
being a meaningful metric (it no longer means the same thing across scenarios that pay
differently), and the only honest number left is:

net worth(t) = ETF portfolio(t) - remaining loan balance(t)

at a fixed horizon, for both scenarios, with identical cash outflow throughout.

The invariant was never the bug. Something else was.

Here's the part that made this finding interesting: our budget invariant was correctly
implemented from day one. Every month's ETF contribution is computed as monthlyBudget - loanRate, not hardcoded, not assumed. If you'd audited that one property in isolation, you'd have
signed off.

The actual failure mode was a level up from that: getting the fair-comparison math right and then
silently comparing two different loans anyway. The panel and the headline agreed on the rule
("budget minus installment goes to the ETF") but disagreed on the input ("what installment,
exactly, on what loan"). A structurally correct comparison of the wrong pair of scenarios is still
wrong, and none of the unit tests around the fair-comparison logic could have caught it, because
that logic was never the part that broke.

The fix threads the same additionals the headline uses through the comparison panel, so both
numbers on the screen are now guaranteed to describe the same loan.

The actual lesson, and something to go check right now

Budget-neutrality gets cited (correctly) as the thing naive pay-down-vs-invest calculators get
wrong. It's necessary. It is not sufficient. You also have to pin every other input, identically,
across both scenarios you're comparing, or the comparison quietly starts answering a different
question than the one on the label, and a green test suite will not tell you.

If you've got a mortgage calculator, a rent-vs-buy tool, or a pay-down-vs-invest comparison
bookmarked somewhere: go feed it a scenario with something extra configured (a lump-sum payment,
an irregular income month, whatever the tool supports), and check whether the comparison panel
still uses it. Most won't tell you either way. Check anyway.

This is the Tilgung-vs-ETF panel in cashcolt, a free
no-signup mortgage calculator hosted in Germany. No tracking, no login, poke at it with your own
numbers.

Your event store is already your audit log

Marc — Wed, 01 Jul 2026 07:51:25 +0000

Your event store is already your audit log

Almost every SaaS I've worked on ends up with an audit_log table. Someone files a compliance ticket — "we need to know who changed what and when" — and a new table appears next to the domain tables. Then the real work starts: writing to it on every mutating endpoint, keeping it in sync, and quietly discovering six months later that three endpoints forgot to log.

That table is a second source of truth. And second sources of truth drift.

What an audit log actually needs

Strip the compliance language away and an audit entry is five fields:

who did it
when they did it
what they touched (which entity)
which action it was
the delta — what actually changed

Plus, for a multi-tenant app: whose data it was, so tenant A can never read tenant B's history.

Now look at what an event in an event-sourced system carries. Every state change is an appended event with createdBy, createdAt, tenantId, aggregateType + aggregateId, type, and a payload holding the delta.

That's the same five fields. The event log already is the audit trail — append-only, ordered, and impossible to forget to write, because writing the event is how state changes in the first place. There's no code path that mutates data without producing an event.

So don't build the table. Query the log.

If the audit trail is already there, the whole "audit feature" collapses into one privileged read over the events table:

export const listQuery = defineQueryHandler({
  name: "list",
  schema: z.object({
    aggregateType: z.string().optional(),
    aggregateId: z.uuid().optional(),
    eventType: z.string().optional(),
    userId: z.string().optional(),
    from: z.iso.datetime().optional(),
    to: z.iso.datetime().optional(),
    limit: z.number().int().min(1).max(100).default(50),
    before: z.string().optional(), // cursor
  }),
  access: { roles: ["Admin", "SystemAdmin"] },
  handler: async (query, ctx) => {
    const p = query.payload;
    const where = { tenantId: query.user.tenantId }; // tenant-isolated at the WHERE
    if (p.aggregateType) where.aggregateType = p.aggregateType;
    if (p.aggregateId)   where.aggregateId   = p.aggregateId;
    if (p.eventType)     where.type          = p.eventType;
    if (p.userId)        where.createdBy      = p.userId;
    // ...time range + cursor omitted for brevity

    return selectMany(ctx.db, eventsTable, where, {
      orderBy: { col: "id", direction: "desc" },
      limit: p.limit,
    });
  },
});

No table, no projection, no write path, no sync job. The filter surface an audit UI wants — by entity, by actor, by action, by time — is just WHERE clauses over columns the events already have.

The two things you still owe

Reusing the event log doesn't come completely free. Two concerns are real:

1. Access control. The event log is the most sensitive read in the system — it's literally everything that ever happened. Gate it hard (Admin / SystemAdmin above) and pin tenant isolation into the WHERE clause itself, not into application logic that a future refactor can bypass. Cross-tenant peeking should be structurally impossible, not politely discouraged.

2. PII. If you dump raw event payloads into an audit view, you'll surface fields you didn't mean to. The clean fix is to strip sensitive values at append time — mark them in the entity definition and never let them into the stored event. Then the audit read physically cannot leak them, because they were never written. Doing it at read time is a filter you'll eventually forget on some new field; doing it at write time is a guarantee.

When this doesn't apply

Honesty: this only works if you're actually event-sourced. If your system does in-place UPDATEs, there's no historical record to query — you genuinely need to start capturing one, and a dedicated table (or CDC/logical decoding off the WAL) is the pragmatic path. This isn't an argument to adopt event sourcing for audit; it's an argument that if you already have it, the second table is redundant.

One caveat even when it fits: event schemas evolve, so your audit reader sees heterogeneous historical payloads. For an audit log that's a feature — you want the exact shape as it was written — but don't mistake it for a clean queryable projection.

The point

An audit log isn't a thing you build. It's a view onto history you're already keeping. If you're appending events, you've been sitting on a complete, tamper-evident audit trail the whole time — the only missing piece was a gated query with the right WHERE clauses.

This is exactly how the audit feature works in Kumiko, a Bun/TypeScript framework where multi-tenancy, GDPR, and audit are bundled features rather than boilerplate you rewrite per project — one ~40-line query handler, no separate table.

How do you prevent AI-generated code from drifting away from your conventions over time?

Marc — Sun, 28 Jun 2026 10:52:37 +0000

We've been generating production features with AI for a while now — auth flows, billing hooks, notification handlers. And we've hit a pattern we don't have a good answer to yet.

The first feature the AI generates looks great. It reads the codebase, picks up the patterns, and the output looks like something a senior dev wrote.

The tenth feature? Less so. Small inconsistencies creep in:

A handler that doesn't follow the error-handling convention
A schema field with a different naming pattern
A test that checks existence instead of behavior

None of it is wrong. All of it is subtly inconsistent.

Fixes we've tried

AGENTS.md / CLAUDE.md — helps, but gets stale and doesn't scale with the codebase

Code review — catches it, but defeats some of the speed advantage

Linting + formatting — catches easy stuff, misses semantic drift

What we haven't solved: giving the AI a "living" representation of your conventions that stays current as the codebase evolves.

We're building Kumiko — an opinionated SaaS framework — partly as an answer to this. If the framework constrains what's possible, drift has less surface area. But I'm not convinced that fully solves it either.

Curious what's actually working for others

Do you just review AI output carefully and accept some drift?
Custom guards / linters that encode your conventions?
Something that auto-generates AGENTS.md from the codebase?

What's your approach?