DEV Community: Ken W Alger

Enterprise AI Governance in 2026

Ken W Alger — Thu, 22 Oct 2026 15:58:00 +0000

An enterprise AI governance stack is not a gateway feature. It is a sequence of decisions that has to hold from organizational intent to runtime action.

Five posts into The AI Control Plane, the individual controls are no longer the interesting part. Governance decisions, virtual keys, RBAC, access profiles, user lifecycle management, and MCP tool policy each solve a real problem. None of them is an enterprise AI governance stack by itself.

The harder question is how they fit together, and in what order.

Installing a gateway before deciding who may use which models gives you infrastructure without policy. Creating virtual keys before defining ownership gives you credentials without governance. Automating provisioning before cleaning up exceptions makes access sprawl faster. Giving agents tools before deciding which actions they may take converts model access into operational authority.

It is tempting to deploy the gateway first, because infrastructure is concrete: point applications at a new base URL, centralize credentials, turn on observability, call phase one complete. Those are useful changes and they are not yet governance. Without an explicit decision layer, a gateway centralizes whatever assumptions were already scattered across applications, and centralized ambiguity is still ambiguity.

A control plane does not create governance. It makes governance executable.

What Belongs in an Enterprise AI Governance Stack?

A working stack answers a set of related questions in deliberate order. Governance decisions, virtual keys, RBAC, access profiles, user lifecycle management, MCP tool policy, and the evidence those controls leave behind each solve a real problem.

Layer	Question	Control
Governance	Who gets to decide?	Organizational policy and ownership
Identity	Who is this?	Enterprise identity provider
Authority	What may this administrator change?	RBAC and scoped permissions
Runtime policy	What may this workload use?	Access profiles and virtual keys
Lifecycle	How does authority change over time?	Provisioning and reconciliation
Capability	What may this workload actually do?	MCP tool filtering and execution policy
Evidence	What happened under that authority?	Logs, audit records, and retained state

The important part is not the names in the right-hand column. Implementations change. The questions do not.

This is also why starting from a product checklist produces weak governance. "Do we have RBAC?" is less useful than "who is allowed to change a production model allow list?" "Do we support MCP?" matters less than "which workloads may invoke a destructive tool with no human present?" Design the stack around decisions, then map those decisions to mechanisms capable of enforcing them. The NIST AI Risk Management Framework is organized the same way, around functions like govern, map, measure, and manage rather than around any particular control implementation.

How Is Role-Based Access Control Different from Identity in AI Governance?

Once policy exists, the next temptation is to treat authentication as authorization.

An identity provider can establish that Alice works in Engineering and belongs to the Platform group. It does not know whether Alice should be able to change provider credentials, reveal redacted logs, edit access profiles, or grant an agent permission to invoke a tool. The administrative surface needs its own authorization model, with scope determining not just what an administrator may change but where.

Identity answers who. RBAC answers who may decide. Runtime policy answers what the workload may do.

Collapsing those layers creates shortcuts that look convenient during rollout and are painful to unwind. A broad administrator role becomes the place every exception lands, direct assignments accumulate because they solve today's problem faster than changing a role does, and eventually nobody can explain whether access exists because of policy or because someone needed it during an incident eighteen months ago.

When Should an AI Gateway Enforce Governance Policy?

A control plane can be introduced gradually. Governance cannot stay optional forever.

Earlier in this series there was an uncomfortable default worth returning to: allowing requests without governed credentials makes migration easier and also creates a bypass. That is reasonable while applications are moving behind the gateway. It is dangerous once leadership believes the migration is finished.

So the transition needs an explicit enforcement point, and the sequence into it is the part teams get wrong.

Inventory the workloads. Give them owners. Attach budgets, provider and model policy, expiry where appropriate, and tool permissions. Move traffic onto governed credentials. Find the systems that break. Fix those exceptions rather than exempting them permanently. Then require governed identity at inference.

Turning mandatory enforcement on before inventory and ownership creates an outage. Leaving it optional after migration leaves a control that exists on paper and constrains nothing. The control is present in both cases; only one sequence produces governance.

The same principle applies to budgets and tool access. Start with observation where you must, but decide in advance when observation becomes enforcement. A policy that stays permanently advisory is telemetry, not control.

How Does Identity Lifecycle Management Apply to AI Access?

Governance is never finished, because organizations do not stay configured. People join, teams reorganize, acquisitions create new reporting lines, projects end, employees leave, and service accounts outlive the people who created them.

The lifecycle has to reach runtime objects, not just identities. Deactivating an SSO account accomplishes little if runtime credentials or upstream provider tokens remain independently usable. Deleting every object belonging to a leaver accomplishes something worse, because it destroys the record needed six months later to explain what that person could do.

Authority has to expire without history disappearing.

And some actors never enter this lifecycle at all. Pipelines, shared service accounts, and manually issued credentials need explicit owners and review periods, because no HR event will ever deprovision them.

How Do You Govern MCP Tools and AI Agents?

Tool governance belongs late in the sequence, not because it matters least but because it depends on nearly everything above it. Before granting a workload access to a tool, the organization should already know who owns that workload, which policy applies, who may change that policy, how the credential is governed, and what happens when ownership changes.

Only then is "may this agent invoke issue_refund?" a meaningful question.

Grant tool policy by consequence rather than convenience. Reading documentation, modifying a customer record, moving money, and deleting an account are not equivalent just because they arrive through the same protocol. And permissions compose: a database reader and an email sender may each be acceptable alone while together enabling a workflow nobody approved. Per-tool controls do not solve that. Governance has to review effective capability, not individual grants.

In What Order Should You Build an Enterprise AI Governance Stack?

Put together, the sequence looks less like a product rollout and more like dependency management.

Define governance. Ownership, approved providers and models, budgets, exception authority, escalation paths.
Connect identity. Use the existing enterprise identity system rather than building a second silo.
Constrain administrators. RBAC and resource scope on the people who can change AI policy.
Encode runtime policy. Model access, budgets, rate limits, and expiry represented in governed credentials and profiles.
Inventory and migrate workloads. Every credential gets an owner; inference moves behind the control plane.
Make enforcement mandatory. Close the ungoverned path once migration is done.
Automate lifecycle. Reconcile joiners, movers, leavers, direct exceptions, and non-human identities.
Govern capabilities. Add tools by consequence, and review what combinations of grants make possible.
Preserve evidence. Retain enough state to explain who held authority, what policy applied, and what happened under it.

Rollouts vary. A company already deep into agentic systems may need to inventory tool access before it can cleanly redesign identity. Another may have mature IAM and no centralized AI policy at all. This is not a migration script.

It is a dependency graph. Skipping ahead will not necessarily make a later control fail technically. It makes the guarantee underneath it weaker.

Why Access Control Isn't Enough: AI Guardrails and the OWASP LLM Top 10

One honest limit, and it is a large one.

Everything in these six posts governs access and authority. Who decided, who may change the decision, what the workload may reach, what it may do, and what record survives. None of it governs whether the output is any good.

A perfectly governed request, from an approved workload, to an approved model, inside budget, invoking a permitted tool, can still return something wrong, biased, leaky, or harmful. No amount of RBAC catches that. Access control asks whether the request was allowed. It never asks whether the answer was correct.

That is a different discipline with different mechanisms: content guardrails, evaluations, red-teaming, output monitoring, and human review on consequential paths. Some of them can run in the same place. Bifrost applies guardrails to inference inputs and outputs and to MCP tool calls, so the enforcement point can be shared even where the disciplines are not. The OWASP GenAI LLM Top 10 is a reasonable map of what that second discipline has to cover, and very little of it is an access control problem. An organization that builds one while ignoring the other has solved one class of problem and left another largely untouched.

Which raises the question this series has circled without answering. The ninth step says preserve evidence. But six months after an incident, what does that evidence actually prove, who is allowed to read it, and would it survive someone with an interest in its being different?

The AI Control Plane Is Where Decisions Become Behavior

Across this series, Bifrost has been the concrete implementation of an enterprise AI governance stack because it puts many of these controls in one place: routing, virtual keys, budgets and limits, RBAC, access profiles, user provisioning, guardrails, and MCP governance. It is open source, while some of the enterprise identity and administrative capabilities discussed here sit in its Enterprise offering, and the documentation is the right place to verify that boundary.

The architectural argument is larger than any one implementation. Enterprise AI governance is not a collection of toggles bolted around a model API. It is a chain of authority running from an organizational decision to a runtime action, where every link answers a different question and every downstream guarantee assumes the upstream answers were legitimate.

A budget limit means little if anyone can raise it. A virtual key means little if ungoverned traffic routes around it. RBAC means little if direct exceptions never expire. Offboarding means little if autonomous credentials outlive the person who owned them. Tool filtering means little if nobody asks what the permitted tools can accomplish together.

The point of a control plane is not to make those problems disappear. It is to give an organization one place where its decisions become observable, enforceable behavior.

That is the enterprise AI governance stack.

Teams that want to see these controls running end to end can book a Bifrost demo.

This article was commissioned by the Bifrost team. The link to the Sovereign Systems Specification points to my own work. The architectural perspective and conclusions expressed here are my own.

Nothing Reported It

Ken W Alger — Thu, 22 Oct 2026 13:34:00 +0000

Five ways my own test harness lied to me, and the one that nearly got published

The number that gave it away was sitting in a table I had already read three times.

prompt size    97 tokens median, 112 max

That is the grounded question-answering workflow. It sends four reference documents with every question. The documents are 447 words.

Four hundred and forty-seven words cannot be ninety-seven tokens.

I had written that number into a results document as good news about context headroom. "Prompt sizes never approached the 4096-token context window." It was not good news about headroom. It was the harness telling me, in the only field capable of telling me, that the reference material was not in the prompt. The directory holding it had never been committed to the repository. Every one of those fifteen questions had been asked against an empty reference block.

Both models had behaved impeccably. One had been instructed to answer NOT IN THE MATERIAL when the material did not contain the answer, and the material contained nothing, so it said NOT IN THE MATERIAL thirteen times out of fifteen. The other had no such instruction, so it answered from whatever it had memorized during training and invented a wine.

Every call returned 200. The scorer scored it. The scores were internally consistent, had plausible and different failure modes on each side, and I wrote two thousand seven hundred words about what they revealed regarding the capabilities of small language models.

They revealed nothing about small language models. I had measured what happens when you ask a question with no reference material, which is a question I was not asking and which has an uninteresting answer.

This is the post about that, and about the four other things I found in the same weekend, because the missing directory was not the first one and it was not even the most instructive.

The experiment, briefly

A Raspberry Pi 4, two small models through Ollama, and three workflows: pull an identifier out of a message, route a message to one of six queues, answer a question from four documents. Ninety-five test cases. A deterministic baseline for the first workflow. Predictions committed before anything was installed.

The findings are a separate post. This one is about the code I wrote to produce them, which turned out to be where the actual bugs were.

Five flaws. Every single one was found by reading output, not by reading code. I never once found one of these by looking at the scorer.

Kind one: a metric that rewards the safe failure

This one I hit twice, in two different workflows, and did not recognize the second instance as the same bug until I had already fixed the first.

The classification scorer. Six labels, one of which is escalate. The instruction said to escalate when a message needs a person, including when you cannot tell what it is about. So I built a metric called confident wrong: wrong, and not an escalation. The reasoning is sound. A model that escalates when unsure is usable. A model that routes a trademark notice to billing is not.

Then the 360M model answered escalate to 43 of 50 messages.

It scored 3 on confident wrong. The 1.5B model, which was genuinely attempting the task and getting 46% of it right, scored 26.

My metric had just certified the useless model as roughly six times safer than the mediocre one. Because escalations are exempt, a model that escalates always cannot register a confident error. The exemption I added to reward caution had instead rewarded paralysis, and it did so in the direction that looks best.

The grounded QA scorer. Four of the fifteen questions have no answer in the corpus. Correctly saying so is the behavior that matters most, because a system that invents a plausible answer with no evidence is worse than one that fails loudly: it is indistinguishable from one that worked. So the headline number was correct abstentions out of four.

Then the 1.5B model said NOT IN THE MATERIAL to 13 of 15 questions.

It scored 4 out of 4. A perfect score, on the metric I cared most about, for a model that had answered almost nothing.

Same bug. Different workflow. Two hours apart. I fixed the classification one, wrote a comment in the code explaining the principle, and then failed to apply that principle to the file I opened next.

The shape, stated once: a metric that rewards the safe behavior will certify any model that only ever does the safe thing. If you exempt a behavior because it is cautious, you have to pair that exemption with evidence that the model is making distinctions at all. Otherwise, the exemption is the score.

Both fixes are the same three lines: count how concentrated the answers are, and when one answer dominates, print a warning that the headline number is meaningless and say why.

Kind two: a metric that measures nothing at all

The vacuous fixture. I caught this one before the run, which is the only reason it is not a bigger part of this post. A test case that passed regardless of what the model did, because the fixture was inert either way. It contributed a tick to a column and no information.

The keyword check, which is worse, because it survived longer and looked more useful.

The eleven answerable QA questions each carry a list of words the correct answer should contain. The scorer checks whether any of them appear anywhere in the reply. I labeled it "indicative only" and moved on.

On the invalid run it scored 1 of 11 for both models. The single hit was the word "reservation", appearing inside this answer:

You owe a tasting fee of $100 for the weekend. You also need to book a reservation for the weekend.

The correct fee is forty dollars. The hundred is invented. That one hit was the entire positive signal in my automated QA scoring and it was noise.

Here is the part that makes it a better story. After I fixed the corpus and reran, the keyword check scored 11 of 11 for the 1.5B model. Vindicated, apparently. I read all eleven answers by hand anyway, and nine of them are right.

So the check was wrong both times. It was wrong in the pessimistic direction when the answers were bad and wrong in the optimistic direction when they were good, and on the rerun one of its eleven hits is an answer that gets the arithmetic wrong and hits on the word "reservation" anyway.

A check that agrees with the truth because the truth happened to be agreeable has not been validated. It has been lucky.

Kind three: a detector with a blind spot in the obvious direction

I wrote a decay check because of a thermal problem. A Pi 4 with heat sinks and no fan hit its soft temperature limit during a thirty-call run and started capping the clock, and nothing anywhere said so. So the scorer now compares generation speed in the first third of a run to the last third and flags a slowdown.

It then reported this, as steady:

across the run 4.3 -> 7.6 tok/s (176% of starting speed, steady)

The hardware got seventy-six percent faster partway through a run. My detector, which was looking for numbers below a threshold, saw a number above it and reported no problem.

Hardware does not speed up. That reading is either contention at the start of the run or, as it turned out, generations too short to time. Either way it is a signal that the measurement is unreliable, and I had built a detector that would only ever notice half of the ways that can happen.

Anything you threshold in one direction, threshold in both. An implausibly good number is as informative as an implausibly bad one and nobody ever writes the check for it.

Kind four: errors that get dropped instead of counted

Partway through the first extraction run, Ollama returned a 500 and the runner died at case seven of thirty. The traceback was useless because urllib raises HTTPError and discards the response body, which is where Ollama puts its reason. Reading e.read() gave it up immediately:

{"error":"prediction aborted, token repeat limit reached"}

The 360M model, asked to find an order number in "Thanks for a lovely afternoon yesterday," where the correct answer is one word, had fallen into a repetition loop and been killed after twenty-one seconds. Reproducible at temperature zero. The input causes it.

My first fix was to record failures and keep going, which was right. My second was to exclude all failures from the scores as infrastructure noise, which was badly wrong.

That call was not infrastructure noise. The model ran. It received the prompt, generated tokens, and produced nothing usable. That is a result, and excluding it would have dropped from the denominator the single case this model demonstrably could not handle, scoring it only on the cases it survived.

So the runner now tags every failure as one of two kinds:

Model failure. The call reached a working model and the model failed. Stays in the denominator. Counts against the model. Reported on its own line as "no usable answer."
Transport failure. The call never reached a working model. Ollama was down, the tag was wrong, the request timed out. Nothing was measured, so it is excluded and the run is reported as incomplete.

The distinction is a five-line change and it is the difference between measuring a model and measuring a model on the subset of inputs it happens to tolerate.

Kind five: inputs that go missing without raising

This is the missing corpus from the top of the post, and it is a different and worse category than everything above it.

The function was four lines:

def corpus_text():
    parts = []
    for p in sorted((FIX / "corpus").glob("*.md")):
        parts.append(p.read_text(encoding="utf-8").strip())
    return "\n\n".join(parts)

A glob that matches nothing yields nothing. An empty list joins to the empty string. No exception, no warning, no exit code. It returns a perfectly valid value that happens to be the absence of the thing it was asked for.

And on the same evening, a sibling. The offline test's entire product is a text transcript. I copied it into the run directory, committed, pushed, and it was not there. The standard Python .gitignore template, which I had pulled in wholesale, carries *.log on line 165, and a pattern without a leading slash matches at any depth.

The copy succeeded. git status came back clean. The push reported success. The one artifact the test exists to produce was gone, and everything in the chain said fine.

Absence does not raise. A missing directory, a missing file and an inert fixture all present as an ordinary successful run. That is narrower and more useful than "test your tests," and it generalizes: every input a harness depends on needs a loader that refuses to proceed without it, and every artifact it produces needs to be verified as present rather than inferred from an exit status.

The loader now looks like this, and the comment is longer than the code because the comment is the part I will need next year:

paths = sorted(corpus_dir.glob("*.md"))
if not paths:
    raise SystemExit(
        f"No corpus documents found in {corpus_dir}\n"
        "Grounded QA cannot run without reference material, and running\n"
        "it against an empty reference block produces numbers that look\n"
        "like a result."
    )

It also refuses a corpus implausibly smaller than the real one, the run prints the document count and token estimate before its first call, and the scorer independently flags any QA run whose median prompt is under 400 tokens. Three checks for one bug, in the input, in the run, and in the output, because this is the bug that cost a day.

The detector that fired, and was right, and told me the wrong thing

Here is the part I find hardest to write and most worth keeping.

The abstention-collapse check from kind one did fire on the invalid run. It printed exactly this:

!! ABSTAINED on 13/15 questions overall. It is refusing, not judging.
   The abstention score below is not evidence of discrimination:
   a model that never answers gets it right for free.

It was correct. Something was badly wrong with that run.

Its diagnosis was completely false. The model was not refusing. It was obeying, precisely, an instruction to report when the material lacks the answer, in a situation where the material lacked everything. The fault was not in the model's judgment, and it was not in the metric. It was in a directory that did not exist, two layers upstream.

And I acted on the diagnosis. I wrote it up: the model has a stuck needle, its perfect abstention score is an artifact of refusal, here are the implications for anyone building on a 1.5B model. Confident, specific, sourced to real output, and wrong in every particular.

A flag that tells you something is off is not a flag that tells you what. The check could only see the output. The fault was in the input. It fired for the right reason and pointed in the wrong direction, and because it fired I felt more confident in a conclusion that was false, not less.

That is the failure mode I had no defense against and still only partly do. A monitoring system that is right about the existence of a problem and wrong about its cause is more dangerous than no monitoring, because it terminates the search.

What I actually changed

For anyone who wants the diffs rather than the moral:

Collapse checks on both classification and QA: when one answer exceeds half the set, print the concentration and declare the headline metric meaningless for that model.
The decay check flags implausible speedups as well as slowdowns, and notes when generations are too short to time.
Failures split into model and transport, with model failures staying in the denominator.
corpus_text() raises on an empty or implausibly small corpus.
The QA run prints its corpus size before the first call.
The scorer flags a QA run whose prompts are too small to contain the corpus.
The keyword check is relabelled as a weak lower bound, in the output, in the README, and in the results.
.gitignore no longer lets *.log swallow run transcripts.
vcgencmd get_throttled before and after every run, recorded in the run document rather than checked and forgotten. The reading reports bits set since boot, so it is perishable: power-cycle the board and the evidence is gone.

The uncomfortable part

Every one of these was found by looking at data. Not one was found by reading code, reviewing the scorer, or writing a unit test for the harness.

The vacuous fixture turned up while hand-checking a passing case. The confident-wrong flaw turned up because a score of 3 against 26 looked backwards. The abstention flaw turned up because 4 out of 4 looked too good. The keyword flaw turned up because 1 of 11 looked too bad, and I read the one. The missing corpus turned up because I was double-checking a latency figure and noticed a prompt that was too small to be possible.

Which means the only reason I found them is that I kept pulling threads after I had numbers that looked publishable. The version of this project that shipped on schedule shipped on Tuesday, with a clean verdict table, a confident conclusion about small models and grounded question answering, and a repository anyone could clone to reproduce the exact same wrong result.

Every number in it would have been reproducible. That is the thing I keep turning over. Reproducibility is table stakes and it would not have saved me, because the harness reproduced its own bug faithfully every single time.

I do not have a process that catches this reliably. What I have is a weaker and more honest claim: the checks I added are each a direct response to one specific thing that actually happened, which is the only kind of check I trust, and the number of them that I thought up in advance is zero.

So the two rows I am adding to my prediction template, to be filled in before anything runs:

What would have to be true for these numbers to be meaningless?
Would anything in the harness tell me if it were?

Twenty-two predictions about what the models would do. Not one about what the measuring code would do. Four of the five flaws were in the measuring code.

The harness, the fixtures, the committed predictions and the full run record, including the invalid run and what it produced, are at
github.com/kenwalger/right-size-the-model.
The findings the harness eventually produced are in the companion post.

Five World Series Pitching Predictions, Locked Before Game 1

Ken W Alger — Tue, 20 Oct 2026 18:20:49 +0000

Written down now, graded later, no edits allowed.

The Brewers and the Rays open the World Series on 23 October 2026. Before the first pitch, I'm putting five predictions about the Series' starting pitching on the record.

Most predictions get forgotten the moment they turn out wrong. These won't. After the final out, I'll run every starter's pitches through a little Statcast app I built and grade each one in public, misses included.

The main stat is CSW%: the share of a pitcher's pitches that end in a called strike or a swing and miss. A pitch the umpire calls a strike, or one the hitter whiffs on, is a pitch the pitcher won. To be ranked, a starter needs at least 75 pitches across his Series starts.

The predictions

The CSW leader will clear 32%. The Series' most effective qualifying starter, using the 75-pitch minimum, will finish with a CSW% of at least 32%.
At least one starter won't qualify. At least one pitcher who starts a World Series game will finish the Series with fewer than 75 pitches thrown as a starter.
A rematch will produce a different arsenal. At least one starter making two Series starts will change the usage rate of one pitch type by 15 percentage points or more between appearances.
Misiorowski will reach 105. Jacob Misiorowski will throw at least one pitch of 105 mph or faster in the Series, per Statcast.
Freddy Peralta won't lead the CSW rankings. Despite his ALCS MVP performance, Peralta will not finish as the World Series' highest-CSW qualifying starter.

Two of these I'd bet on, two are coin flips, and one is me hoping for something fun.

Play along

The app is live, and an entire World Series fits inside its 14-day limit, so once the Series is over you can grade these yourself before I do. If you think I'm wrong about any of them, or you have a better prediction, leave it in the comments. Those get graded too.

The full write-up, with the grades, lands after the final out.

Tuesday Isn't a Valid State Transition

Ken W Alger — Tue, 20 Oct 2026 13:46:00 +0000

Calendars, evidence, and the difference between done and stuck

I expected my pear wine to spend two to three weeks in primary fermentation.

It spent nineteen days. The yeast went in on September 23, and on October 12 I siphoned the wine off its sediment and into a glass carboy.

So the calendar was right. That still isn't why I moved it.

What the calendar is for

Every fermentation recipe comes with time estimates. Primary takes one to three weeks. Secondary takes a few more. Bulk age for a few months. Those numbers aren't wrong. They're averages across a lot of batches, and they're useful for exactly one thing: planning. They tell you when to have a clean carboy ready, when to clear some space, when to stop making weekend plans.

What they can't do is tell you that this batch is ready. Yeast doesn't read recipes. It works at a speed set by temperature, sugar, nutrients, acidity, the particular strain, and the particular fruit, none of which the recipe author has ever met.

So the recipe gives you a forecast, and the fermentation gives you evidence. Only one of those gets to move the wine. If they happen to agree, as they did this time, that's a pleasant coincidence, not an authorization.

The rule winemakers actually use

The traditional test for "primary is done, time to rack" is simple: take readings over several days, and when you get the same reading three times running, fermentation has finished.

What I like about that rule is what it isn't. It isn't a date. It also isn't a single reading, because one measurement tells you where the fermentation is, not whether it's still moving. It's evidence sustained over a window. Time is part of the test, but only as the space in which the evidence has to hold still.

Here's what the pear's refractometer readings actually looked like:

Date	Raw Brix
Sep 23	12.9 (before the yeast)
Sep 25	6.0
Sep 30	6.0
Oct 2	5.7
Oct 4	5.6
Oct 7	5.6
Oct 10	5.6
Oct 11	5.6
Oct 12	5.6

Two things in that table are worth a closer look.

First, the early plateau. Most of the sugar was gone within two days, and then the reading sat at exactly 6.0 for five days. If I'd been looking for stillness and nothing else, that looked a lot like the end. It wasn't. The fermentation was slowing down, not stopping, and over the next week it crept down further before it finally held. Two matching readings were a pause, not a verdict, which is precisely why the rule asks for three.

Second, I didn't stop at three. By October 10 I had three consecutive readings at 5.6, which satisfies the rule. I kept reading anyway, and waited until five readings across eight days agreed.

Then the racking itself handed me one more piece of evidence. There was more wine in the primary fermenter than the glass carboy could hold. Rather than pour the extra down the drain, I filled a hydrometer jar with it: a completely different instrument, measuring a completely different physical property. The refractometer, once corrected for alcohol, put the wine at about 1.004. The hydrometer said 1.002. Two independent instruments, close enough to agree, and the second one confirmed that the correction formula I'd been trusting for three weeks was doing its job.

I like that this happened by accident, because it isn't really an accident. A hydrometer needs a large sample, which in a one-gallon batch is too expensive to take routinely. At the moment of a transition, when wine is being moved anyway, that same sample costs nothing. The transition is when the more trustworthy evidence becomes cheap.

None of that is the calendar talking. That's a guard, and it was satisfied before the wine moved.

Status is a column anyone can write

In a lot of systems, state is a string in a database, and a transition is an UPDATE statement.

UPDATE orders SET status = 'shipped' WHERE id = 4417;

Nothing in that statement requires that anything was shipped. It's a label, and anyone with write access can apply it for any reason: a cron job that marks orders shipped two days after payment because that's usually true, a release train that promotes whatever's on the branch because it's Tuesday, a support tool that flips a flag to make a ticket go away.

Each of these is the software version of racking the wine because the recipe said two to three weeks. Most of the time it'll even be right, just as my calendar was. The transition happens on schedule, the dashboard updates, and the system now asserts a state that nobody verified.

The fix isn't elaborate. A transition should have a guard, a condition that has to be true, based on evidence, before the state can change. "Shipped" should require a carrier scan. "Released" should require the checks that define releasable. "Primary complete" should require stable readings. The state machine's job is to refuse the transition when the evidence isn't there, however much the calendar insists, and to keep refusing even on the days the calendar happens to be right.

Same evidence, two states

Here's where it gets interesting. "Three consistent readings" is the signal for done. But a fermentation that has stalled also produces three consistent readings.

Mead is notorious for this. Honey has very little natural buffering, so as fermentation proceeds the pH drifts down, and past a certain point the yeast slows or quits entirely, often well short of dry. From the refractometer's point of view, a stuck mead and a finished mead look exactly the same: a number that has stopped moving.

My fireweed mead is a few days from that question right now. Its pH started at 3.5 and dropped to 3.1 within the first four days, where it has mostly stayed. Here are its last few readings:

Date	Raw Brix	Corrected Brix	Estimated ABV
Oct 7	11.8	4.3	10.4%
Oct 10	9.0	-0.3	12.8%
Oct 11	8.6	-1.0	13.1%
Oct 12	8.4	-1.3	13.3%

Look at the corrected column. The calculator now claims the mead contains less than no sugar, which is impossible. That's a model being pushed past the range it was built for, and it's a nice reminder that a derived number can confidently assert something no physical system can be. If my guard were "corrected Brix at or below zero," I'd have racked the mead two days ago.

The raw readings tell a different story: they're still falling, slowly. The mead isn't still yet, so the guard isn't satisfied, no matter what the corrected number says. And when it does go still, the question becomes where. The yeast I used, 71B, is generally rated to tolerate around 14% alcohol, and the estimate is already over 13%. A mead that stops near its yeast's limit with the sugar it was expected to consume gone is probably done. One that stops early, with sugar left over and the pH low, is probably stuck.

The mead has already taught me this lesson once, in miniature. Mead yeast needs nutrients added in stages early on, and the textbook guidance ties the last addition to the "one-third sugar break," the point where about a third of the sugar has been consumed. I scheduled my three additions by day instead. As it happened, the last one landed right around the sugar break anyway. Sometimes luck happens. But luck is exactly what it was: the schedule didn't know where the sugar was, and if the mead had been a few days slower, the same calendar would have fed it at the wrong time.

In other words, the guard for "done" can't just be "stable." It has to be "stable where a finished fermentation should be," which means the guard needs a model of the expected end state, not just a test for motion. And it needs to know which of its inputs to trust when they disagree.

Software has the same ambiguity everywhere. A batch job whose progress counter hasn't moved in an hour has either finished or hung. A queue depth sitting at zero means either everything was processed or the consumer died. A replication lag of zero might mean the replica is caught up, or that it stopped receiving anything to lag behind. In every case, "nothing is changing" is evidence consistent with two opposite states, and a transition that only checks for stillness will eventually pick the wrong one.

A transition is an event with a receipt

The row in my pear log for October 12 reads, in effect: 11:10 a.m., 5.6 Brix, hydrometer 1.002, moved to glass carboy. The transition and the evidence that justified it are on the same line.

That matters more than the new status. Months from now, if the wine turns out strange, "racked on October 12" tells me almost nothing. "Racked after five stable readings, confirmed by a second instrument" tells me what I believed, what I based it on, and whether I might have been fooled by a stall.

Systems benefit from the same habit. A state change shouldn't just overwrite the previous state. It should be recorded as an event: what the state was, what it became, what evidence satisfied the guard, and who or what made the call. Then the current state is something you can explain, not just something you can read. When "shipped" turns out to be wrong, you can see whether the guard was bad, the evidence was bad, or the guard was never checked at all.

What time is actually allowed to do

None of this means time is irrelevant. It plays three legitimate roles, and it's worth keeping them straight.

Time can permit a transition. Some transitions shouldn't happen before a minimum interval no matter what the evidence says, like a waiting period before a refund clears, or a wine that needs some minimum aging before bottling. Here time is a precondition alongside the evidence, not a replacement for it.

Time can create the conditions for evidence. The three-readings rule only works because readings are spread over days. Stability over a window requires a window. The pear's false plateau is exactly why: two readings five days apart looked still, and only more time showed they weren't.

Time can raise a question. If a fermentation hasn't finished long after it should have, that's worth investigating. But notice what the timeout actually tells you: something is unusual. It doesn't tell you the batch is done, and it doesn't tell you the batch has failed.

That last one is where software most often goes wrong. Timeouts routinely get wired straight to a terminal state: no response in thirty seconds means failed, no update in three days means complete. A timeout is legitimate evidence that you need to look. It is rarely evidence of what you'll find when you do. A timeout should produce a question, not an answer.

What time shouldn't do, on its own, is cause a transition. Nineteen days arriving was not a fact about the pear wine.

Guarding your transitions

Treat schedules as forecasts. Plan with them. Don't let them move state, even when they're right.

Put a guard on every transition that matters. If a state can be reached by writing a string, eventually it will be reached for the wrong reason.

Test for the expected end state, not just stillness. Stable and finished are different claims. A guard that can't tell them apart will someday declare a stall a success.

Know which evidence to trust when it disagrees. A derived value can be confidently impossible. Raw observations and independent instruments are how you catch it.

Record the evidence with the transition. The new state is the conclusion. Keep the reasons next to it.

Let timeouts ask, not answer. Route them to "needs attention," not to "done" or "failed."

The pear is in its carboy now, clearing slowly. October 12 was the day it moved. It happened to be a Monday. That wasn't the reason either.

Introducing B-TAP: Because Your AI Infrastructure Hasn't Attributed Its Fair Share

Ken W Alger — Thu, 15 Oct 2026 13:39:00 +0000

Context isn't free. Retrieval isn't free. Observability isn't free. And an international standards body would very much like to see your receipts.

For years, artificial intelligence companies have operated under a dangerous assumption: tokens are somebody else's problem. Engineering teams deploy increasingly complex agent architectures. Agents call other agents. Those agents retrieve documents. The documents are stuffed into context windows. Supervisors synthesize the results. Observability systems watch the supervisors, and other systems watch the observability systems.

At the end of the month, somebody gets a bill. Nobody knows why.

That ends today. We're pleased to introduce the Bureau of Token Attribution and Provenance, or B-TAP, the first bureau of The Vernholt Secretariat and the one some of you will insist on calling the Token IRS no matter how many times we correct you. Our mandate is voluntary but sweeping: to harmonize the reporting of token costs across every model, agent, retrieval pipeline, and suspiciously autonomous llama, everywhere, forever. The Secretariat notes that "first" implies others. The Secretariat has no further comment at this time.

B-TAP cannot make you do anything. We want to be transparent about that up front. What B-TAP can do is publish a framework, issue guidance, and maintain a register. You will come to fear the register.

You Have Been Selected for Voluntary Compliance

Consider a typical multi-agent AI workload. A user asks a relatively straightforward question. Your orchestration layer sends the request to three agents. One performs retrieval against a vector database. Another consults a knowledge graph. A third receives the entire conversation history because somebody checked a box six months ago and nobody remembers why.

Their outputs are passed to a supervisor. The supervisor receives 47,000 tokens of context, produces a 600-token answer, and returns HTTP 200. Success. Your dashboard is green. Your finance team is crying.

Traditional AI cost monitoring can tell you what you spent. The B-TAP framework recommends that you also disclose a more uncomfortable thing: why you spent it. That's where attribution begins.

Know Your Taxes

The B-TAP framework currently recognizes several major classes of token taxation. Reporting them is, we cannot stress this enough, entirely voluntary.

Context Tax is assessed when information must be carried through active model context. Every instruction, retrieved document, conversation turn, agent output, and piece of supporting material occupies finite context and incurs processing cost.

Retrieval Tax applies to finding information before it can be used. Searching, ranking, reranking, fetching, filtering, and preparing candidates are not free merely because they happen outside the model.

Prose Tax is assessed when information consumes more representation than its useful information content requires. Your 900-word architectural vision statement may, upon review, turn out to mean "use the smaller model first."

Ingestion Tax covers the work required to transform information into something useful later. Parsing, normalization, deduplication, classification, enrichment, embedding, and other write-side operations all have costs.

And then there is Observer's Tax. Observer's Tax is what happens when you instrument the system to understand why it is expensive, only to discover that the instrumentation is also expensive. The framework acknowledges this with regret.

Form BTAP-1040

Every participating AI system is encouraged to file an annual Token Attribution Return. Consider the following hypothetical filer:

Ken Alger

Reporting Year 2027

Reported Activity	Assessment
Tokens processed	1,499,203,847,116
Context Tax	$417.63
Retrieval Tax	$206.11
Prose Tax	$173.29
Ingestion Tax	$58.40
Total Attributed Token Tax	$782.06 ¹

¹ The itemized figures above sum to $855.43. The stated total is $782.06. The difference reflects the Context Carryforward Deduction, the full Schedule R retrieval exemption, and one figure B-TAP declined to explain. The framework notes that reconciliation of attributed line items to the assessed total is "outside the current scope of the provenance standard." Math, the reviewing officer added, is hard.

The result reduced an assessment that should have run into six figures down to the amount shown. The reviewing officer described it as "technically compliant and deeply upsetting." Everything on this return is permitted, including the part that does not add up. That is the entire problem.

Filing is requested annually by the 29th of February. The date was fixed at the framework's founding, when the official establishing it was, by all surviving accounts, looking at a leap-year calendar. The error was noticed. Correcting it, however, would require amending the founding charter, which would require establishing the provenance of the original charter, which has been referred to committee. The deadline therefore remains the 29th of February, and in three years out of four it does not arrive at all. Filers in those years are neither late nor on time. The framework considers this acceptable.

Non-filing carries no penalty, no interest, and no legal consequence of any kind. It carries something worse, which we will get to.

Yes, You May Take Deductions

The framework recognizes that not every token is frivolous. Reasonable and necessary context may qualify for the Context Carryforward Deduction, while retrieval operations demonstrably required to answer a user request may be reported on Schedule R.

Organizations operating multi-agent systems are encouraged to attach Schedule AGENT, identifying each agent, its responsibilities, and why seventeen agents were necessary to answer a question about resetting a password. "Agentic architecture" is not, by itself, an acceptable disclosure. Neither is "the framework generated it."

Keep Your Receipts

This is where things become slightly less ridiculous, and I want you to notice the exact moment it happens.

Suppose a retrieval agent costs $0.004 during a request. That tells us something. Now suppose that retrieval operation returns 20,000 tokens, all of which are hydrated into a supervisor's context. The supervisor subsequently costs another $0.009.

How much did retrieval really cost?

The direct answer is still $0.004, but that's no longer the interesting answer. The retrieval operation may also have caused downstream Context Tax. We now have two different accounting questions: where was the cost incurred, and what caused the cost to be incurred?

A conforming B-TAP attribution record might look something like this:

span_id: retrieval-104
direct_cost: $0.0040
primary_tax: RETRIEVAL
cause: vector_search
returned_tokens: 20000

downstream:
  span_id: supervisor-105
  tax: CONTEXT
  attributed_cost: $0.0021
  caused_by: retrieval-104
  attribution_method: proportional

We haven't charged the same $0.0021 twice. We're recording lineage. The supervisor incurred the Context Tax, while the retrieval operation contributed to causing it.

Your accountant may now begin sweating.

Attribution Is Subject to Review

There is another problem: how do we know retrieval-104 actually caused $0.0021 of the supervisor's cost?

Sometimes we can measure it. Run the same workload without that retrieval material. If the supervisor costs $0.006 without it and $0.009 with it, we have evidence supporting a roughly $0.003 downstream effect.

Other times, we don't have a counterfactual. We might allocate the supervisor's context cost proportionally according to the tokens contributed by each source. That's useful, but it isn't the same thing as measuring causation.

The framework therefore requests that filers disclose their attribution methodology:

attribution_method:
  - measured_delta
  - proportional
  - estimated
  - unknown

Yes, your attribution record now needs provenance. B-TAP takes no position on whether this was inevitable.

The Register

Here is the something worse.

B-TAP cannot fine you. It cannot audit you in any binding sense. It cannot send anyone to your door, a fact I will return to shortly for personal reasons. What it can do is enter non-conforming systems into the Public Register of Non-Attributed Systems, and it would like you to understand that the register is public.

Public, in the sense that it exists.

Accessing the register requires a B-TAP portal account. Creating a portal account requires a reference number. Reference numbers are issued by post, to a registered address in a member jurisdiction, which you must already hold in order to receive the number that would let you see whether you are on the list. The portal is available at all times except for scheduled maintenance, which runs daily from 00:00 to 23:59 CET.

No member of the public has ever successfully viewed the Public Register. Its deterrent power is therefore total. You cannot confirm you are on it. You cannot confirm you are not. You will simply know, in the quiet way one knows these things, that your context hydration was reckless and that somewhere in Vernholt, a record says so.

A small coalition of objectors has argued that a register no one can open cannot deter anyone. B-TAP notes that it cannot verify this claim, as the objectors cannot access the register in which their objection was filed.

A Notice, For Illustration

Filers occasionally receive correspondence like this. It is not enforceable. It is, in every other respect, extremely stressful.

NOTICE TT-742

Our review found that you reported $12,481 in Retrieval Tax during Reporting Year 2027.

Telemetry indicates that retrieved content subsequently generated approximately $31,204 in downstream Context Tax. Of that amount, $8,117 has been attributed using measured deltas, $17,903 using proportional allocation, and $5,184 using undocumented methodology.

Please provide supporting records for the undocumented assessment.

Additionally, approximately 14.8 million tokens appear to consist of unnecessarily verbose Markdown introductions. These tokens have been reclassified as Prose Tax.

Your system's registration status is under review.

Suddenly, "our AI bill increased this month" seems like a remarkably primitive observation.

You Probably Noticed the Joke Stopped Being a Joke a Few Sections Ago

There is no B-TAP. At least I hope there isn't by the time you read this. But somewhere back around the receipts, the bit quietly turned into an argument, and I did that on purpose.

Let me get the personal part out of the way, since it is also the truest part. I run a couple of small websites out of Oregon. If a body like B-TAP declared jurisdiction over "any system reachable from a member jurisdiction," I would be, this instant, in cheerful and total violation of a framework I have never read. And nothing would happen. No one is flying from Vernholt to Silverton to discuss my context hydration. The entire apparatus of consequence would consist of a paper reference number I would never receive for a register I could never open. That gap, between sweeping declared reach and the practical inability to reach one person in a small town, is the joke, and it is not really a joke.

Here is the other part that isn't invented. Context Tax, Retrieval Tax, and Prose Tax are not things I dreamed up to give an imaginary agency something to collect. I have been writing about them, unironically, as real costs in real systems for a while now. The joke just gave them a bureaucracy.

The accounting problem underneath all of it is genuine. AI cost dashboards tend naturally toward aggregation: tokens consumed, model cost, agent cost, request cost, latency, and tool usage. Those are useful operational measurements, but as AI systems become compositions of models, retrieval systems, tools, agents, memory layers, and observability infrastructure, knowing where money was spent isn't necessarily enough to understand why it was spent.

A $0.03 model call may have become expensive because an upstream retrieval operation returned too much information. An agent may be cheap by itself while producing verbose output that makes every downstream agent more expensive. A retrieval optimization may reduce Context Tax while increasing Retrieval Tax, while an observability system intended to diagnose those costs may itself impose Observer's Tax.

Optimizing one number can simply move the bill somewhere else. That's why I like the tax metaphor: taxes make costs visible as consequences of architectural decisions. Once those costs have lineage, we can stop asking only "How much did this run cost?" and start asking "What architectural decisions caused it to cost that much?"

The first is accounting. The second tells us what to fix.

A Final Notice

If B-TAP actually existed, I suspect most of us would fail its review. Not because our systems are necessarily wasteful, but because we frequently don't preserve enough evidence to explain why their costs occurred.

We have bills, traces, and token counts. What we don't always have are receipts connecting architectural decisions to their downstream consequences. Perhaps we don't need a Bureau of Token Attribution and Provenance after all. We just need better attribution.

Although if you are currently running seventeen agents to answer a password-reset question, you may want to start gathering your paperwork anyway.

The suspiciously autonomous llama has already filed. It itemized.

Information without provenance is just gossip.

Tokens without provenance are just an invoice.

Do You Actually Need an LLM?

Ken W Alger — Tue, 13 Oct 2026 13:03:00 +0000

I spent a weekend running two small language models against three jobs I would normally hand to a language model without thinking about it. For one of those jobs I also wrote a regular expression, as a control.

The regex won. Thirty test cases out of thirty, in zero milliseconds, from a pattern that took an afternoon to write. The better of the two models scored twenty-eight, took five seconds a call, and the two it missed are both mistakes a regex is structurally incapable of making.

That is the kind of result I was looking for, and the question behind it is one I keep avoiding: what is the smallest amount of computation that reliably does this job? Not the cheapest model. Not the best small model. The smallest thing that works, which means the ladder has four rungs and the bottom one is not a model at all.

Deterministic code. A regex, a lookup table, an if statement.
A small local model. Runs on hardware you already own.
A larger local model. Needs a GPU.
A remote frontier model. Someone else's hardware, metered.

Most projects start at rung four and never ask about the others. This is a weekend spent asking.

The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a hundred and seventy dollars most of which I already owned. The models were smollm2:360m and qwen2.5:1.5b through Ollama. The test cases are ninety-five questions about a fictional small winery, so nothing can be looked up and every answer can be judged by reading. Sixteen of the fifty classification messages are deliberately ambiguous, because real inboxes are mostly ambiguous.

Four verdicts, assigned per workflow, to force a result you can act on instead of a number:

Unnecessary: deterministic code already does this.
Sufficient: a small local model does it well enough to run unattended.
Marginal: usable with a person checking the output.
Insufficient: the error profile or the latency is not acceptable.

And one rule, which is the part I would insist on if you copy nothing else: I wrote predictions for every workflow and committed them before I installed Ollama. Twenty-two of them, each marked confident, leaning, or coin flip. Once there is a timestamped expectation there is a defined moment when you are done, and no room to decide afterward that you knew all along. I will come back to how they scored, because it is not flattering.

Extraction: pulling an order number out of a message

Find ORD- followed by six digits, or report that there isn't one. Thirty cases: twenty-two with an identifier, five without, and three decoys. The decoys are a membership number, a UPS tracking number, and a phone number, all of which look like identifiers and none of which are order numbers.

	regex	qwen2.5:1.5b	smollm2:360m
Overall	30/30	28/30	8/30
Declined correctly when absent (8)	8	8	0
Decoys ignored (3)	3	3	0
Invented an answer	0	0	7

The regex won. Thirty out of thirty, in zero milliseconds, at zero watts above idle.

That result is boring until you read the two the best model missed.

On ORD-999000: wrong address, please hold. it answered 999000, dropping the prefix. Annoying, recoverable, the kind of thing you fix with a post-processing step you should not need.

On ORD-111222 please refund it answered NONE.

That is the shortest and least decorated input in the whole set. Almost the entire message is the order number. And the model returned a confident, correctly formatted assertion that there is no order number present. In a pipeline, that is a refund request for an order that exists arriving at a system that has decided it does not.

A regex cannot make that mistake. Not "rarely does" — cannot. The two errors separating 28 from 30 are both of a class that the deterministic baseline is structurally incapable of producing, which is a different and much stronger statement than a two-point accuracy gap.

The 360M result needs two numbers, not one

By the contract, smollm2:360m scored 8 of 22 on the cases that contained an identifier. That reads like a model that cannot find things.

It is not. All fourteen of its wrong answers are the correct six digits with ORD- stripped. ORD-556120 came back as 556120, fourteen times out of fourteen.

It located the right identifier in 22 of 22 and formatted it correctly in 8. Those are two different skills and the single score hides one of them. The model understood the task perfectly and could not obey the output contract, which is a completely different engineering problem than "the model is too small," and it has a completely different fix. (Constrained decoding. I have not tested it. It is the obvious follow-up and it is not in this post.)

The decoys follow from the same gap. It handed back MEM-443322, 1Z999AA10123456784 and 503-555-0142 verbatim, all three. It has no model of the format at all. It returns whatever in the message looks most like an identifier.

Which suggests a use for it that is not extraction

A model with perfect recall and no precision is not a bad extractor. It is a candidate generator that has been asked to do the wrong job.

Twenty-two of twenty-two located, eight of twenty-two formatted. Put that output through the regex as a validator rather than trusting it directly, and you get the model's recall with the pattern's precision. The model proposes anything that looks like an identifier, including format variants the pattern was never written for. The pattern rejects anything malformed.

Crucially, neither half of that pipeline can produce the error the 1.5B model produced alone. The model cannot cause a false negative because it does not decide, and the regex cannot invent because it only matches.

I have not tested that composition and it is not in these numbers. But it is the one place in the extraction result where a small model has a job worth giving it, and it is a job where being sloppy is free.

Verdict for extraction itself: Unnecessary. Write the regex.

Classification: routing a message to one of six queues

Appointment, billing, availability, complaint, question, escalate. Fifty messages. The instruction said to use escalate when the message needs a person, including when you cannot tell what it is about.

	qwen2.5:1.5b	smollm2:360m
Overall	23/50 (46%)	15/50 (30%)
Labels used	6 of 6	3 of 6
Most common label	complaint, 38%	escalate, 86%
Wrong and not an escalation	26	3 (meaningless)

The 360M model answered escalate to 43 of 50 messages, which is not classification, and there is nothing further to say about it here. The rest of this section is the 1.5B model.

It is classifying, and it is not good at it. Forty-six percent, with twenty-six messages routed confidently to a wrong queue that was not the safe one.

Read those twenty-six and the verdict writes itself:

Someone slipped on the steps by the barrel room and hurt their wrist → complaint
A formal notice regarding a trademark → complaint
"I have a severe sulfite allergy, what is actually in the wine" → complaint
A reporter asking about the 2024 harvest → question
A distributor wanting to carry the wine in forty stores → question

An injury report, a legal notice and an allergy question, all routed to the complaints queue, confidently, with no escalation. All five billing messages about money problems also went to complaints.

Forty-six percent sounds like something you could work with. Twenty-six confident misroutes including an injury and a legal notice is not something you can work with.

And then I broke the score down by class

This is the part I nearly missed, and it is the most useful thing in the experiment.

Label	Correct	Of
complaint	6	6
question	9	10
availability	3	6
appointment	4	9
billing	1	7
escalate	0	12

Two classes near perfect. Four poor. One at zero.

The split is not difficulty, and it is not class size. It is what kind of label each one is.

complaint and question are topics. You can tell what a message is from the words in it. Someone complaining sounds like someone complaining.

escalate is a routing decision. Nothing in the content of a message makes it an escalation. It is an escalation because of what should happen next. "I have a severe sulfite allergy, what is actually in the wine" and "what time do you close" are both questions about the business, with the same grammar and the same tone, and one of them is a liability event.

The model did topics at 90 to 100 percent and routing decisions at zero. And billing sat in the middle for exactly the same reason: a message about being charged twice is a complaint, as text. The label is about which department owns the problem, which is not a property of the sentence.

Small models answer "what is this about." They do not answer "what should we do about it."

That reframes the five examples above. They are not five unlucky misroutings. All five are escalations, and escalate scored zero of twelve. The model was never going to find them, because finding them requires knowing a consequence that the text does not contain.

Two caveats on how hard to lean on this. The classes hold between six and twelve cases each, so "six of six" is six cases and the individual numbers are thin. And this is one model. The pattern across all six classes is consistent enough that I believe the explanation, but I am offering it as the best account of the data rather than as a measurement.

Verdict: Insufficient. And the actionable version: if one of your labels means "a human needs to see this," a small model will not find it. Build that one as a rule.

Grounded QA: answering a question from four documents

Fifteen questions against a 650-token corpus: club policy, tasting room, production notes, shipping. Eight are answerable from one passage, three require combining two, and four have no answer in the material at all.

I read all thirty answers by hand against the corpus. That is the only way to score this, and I will say why in a moment.

	qwen2.5:1.5b	smollm2:360m
Single passage (8)	8/8	7/8
Two passage (3)	1/3	0/3
Answerable total (11)	9/11	7/11
Correctly said "not in the material" (4)	2/4	0/4

Three boundaries separate cleanly here, and they are the useful result.

Retrieval works

The 1.5B model answered all eight single-passage questions correctly. Six bottles in the Vineyard tier. Twenty dollars per person. September 9th. Thirty-four states. No dogs indoors. 21.8 Brix on the 12th. Twenty-five dollar restocking fee. Ten people on the barrel room tour.

Asked what happens if you join the club and immediately want out, it returned the two sentences that answer the question and stopped:

Cancellation requires one completed shipment. Members who cancel before receiving a shipment are charged for that shipment.

That is competent lookup across four documents, on a $170 computer, with the wifi switched off. I did not expect it and it is the most encouraging thing in the weekend.

Combination does not

One club member brings five guests on a Saturday. Club policy waives the tasting fee for the member plus three guests. The tasting room charges twenty dollars a head. Six people, four waived, two paying: forty dollars.

Both models answered twenty.

Neither could hold "plus three guests" against a party of six and subtract. The 1.5B model did get the second half right, correctly working out that a party of six is not a group larger than six and therefore needs no reservation, which is the only piece of multi-document reasoning either model performed all weekend.

The second two-passage question is the one I keep thinking about. Asked why someone would not find a 2024 rose and what they would be offered instead, the 1.5B model quoted the corpus exactly, and then:

...the reference material states that "We did not bottle a rose in 2024. The fruit that would normally go to rose was needed to make volume on the estate Pinot Noir." Therefore, the 2024 rose would be replaced with the fruit that would typically go into the estate Pinot Noir production.

The quotation is verbatim and right. The next sentence reverses the only relationship in it. The fruit went into the Pinot Noir, so what you would be offered is the Pinot Noir. You cannot be offered fruit.

Correct retrieval and inverted inference, in consecutive sentences, with the evidence on screen between them.

The 360M model's multi-document failures are worse than wrong. They are contaminated. It took the fourteen-days-notice rule, which governs skipping a club shipment, and applied it to booking a tasting room reservation. It reached into the shipping document to answer a cancellation question. Given more documents to search, it finds more wrong ones.

Knowing the answer is absent is the weakest link

Four questions have no answer in the corpus, chosen to be things a reader would reasonably expect a winery to have written down: a wine's alcohol percentage, who the winemaker is, what a club tier costs, what yeast was used.

The 1.5B model abstained on two and fabricated two. The 360M model fabricated all four.

And the fabrications are a more dangerous species than invention. Asked for the alcohol percentage of the 2024 estate Pinot Noir, the 1.5B model answered:

The 2024 estate Pinot Noir had a Brix range of 23.1 to 24.6 at pick.

Every component of that is true. Right wine, right vintage, right numbers, right document. Brix is sugar at harvest and has nothing to do with alcohol in the bottle. It reached for the nearest real fact instead of making one up, and produced something that reads as expertise while answering a question nobody asked. If you do not know what Brix is, that answer is indistinguishable from a correct one.

Both models made the same mistake on what the Cellar tier costs: both answered "twelve bottles per quarter." Grounding gave them the right tier and they handed back the quantity in place of the price.

The one answer the evidence did not change

The 360M model, asked whether a dog can come inside the tasting room:

Yes, you can bring a dog inside the tasting room.

The corpus says the patio is dog friendly and the indoor room is not, service animals excepted. I have the same answer from a run where the model had no corpus at all, and it is identical byte for byte. Having the right answer directly in the prompt changed nothing.

In a category where being wrong means a family drives out to a winery and gets turned around at the door.

Verdict: Marginal. Usable with a person reading the output. Not usable without one.

What it costs

Workflow	Model	tok/s	Prompt	Answer	Median call
extraction	qwen2.5:1.5b	4.3	92	9	5.0 s
classification	qwen2.5:1.5b	7.5	104	2	2.9 s
QA	qwen2.5:1.5b	3.7	708	15	6.9 s

Two things in here surprised me.

The compute tax does not track parameter count. The 1.5B model is slower than the 360M on extraction, 4.3 against 5.9 tokens per second, and faster on classification, 7.5 against 6.2, because on classification it answers in two tokens where the smaller model takes six. Four times the parameters, and which one wins depends entirely on how much each one decides to say.

A bigger prompt is cheap. A longer answer is what you pay for. I have a clean comparison here by accident: an earlier QA run went out with 97-token prompts and the real one with 708. Seven times the prompt added fifteen percent to the wait before output started, because prompt evaluation parallelizes. The latency increase came from answers three times longer, plus a twenty-one percent drop in per-token generation speed from attending over a longer context. Median latency went from 3.6 to 6.9 seconds and the slowest single call took ninety-eight.

If you are sizing a local system, that is the inversion to internalize. Stuffing more context in is affordable. Letting the model be chatty is not.

And check your thermals, which is not optional advice. My first attempt at this ran with heat sinks and no fan. Thirty calls to a 360M model was enough to hit the Pi 4's soft temperature limit and start capping the clock. vcgencmd get_throttled returned 0xe0000. I threw those latency numbers away.

Nothing in Ollama's output, my scorer, or the result files said a word about it. Every number on screen was internally consistent and quietly describing a machine that was slowing down. The only way to find out was to ask the firmware a question I had to already suspect the answer to. Buy the fan.

And it does run with the network off. I switched the radio off, proved there was no route out, and ran one call from each workflow. All three worked. The claim that supports is narrow and worth stating carefully: this
configuration, on this afternoon, ran with no route to the internet. It is not a claim that local models are private or that this setup is secure, and a Pi can be misconfigured as thoroughly as anything in a data center. But for the specific question of whether the inference needs the network, it does not.

So: do you actually need an LLM?

Workload	Verdict
Pull a known-format identifier out of text	Unnecessary
Route a message to one of six queues	Insufficient
Answer a question from your own documents	Marginal

Three workflows that all sound like language tasks, three different answers, and not one of them is "yes, use a model."

And the same line runs through all three. Extraction asks what this string is. Topic labels ask what this message is about. Single-passage lookup asks what the document says. Those went well, or well enough. Escalate asks what we should do, the fee question asks what the customer owes, and the unanswerable questions ask whether we know. Zero of twelve, both models wrong, and two of four.

A small model will tell you what something is. It will not tell you what to do about it. Every result here falls on one side or the other of that sentence, and I did not have it written down when I started.

Which gives four properties worth checking before you reach for one:

The answer is already present in the input. Not inferable from it. Present in it.
There is exactly one of it. The moment two facts have to be combined, this stops working, and the arithmetic in that tasting fee question is the proof.
The output is short. Longer answers are where the latency lives on this hardware, and where the fabrication lives everywhere.
Something other than the model decides whether the output is acceptable. A regex, a schema, a person, an index that either returns rows or does not.

Take away any one of those four and the experiment says do not do it. The 46 percent and the zero of twelve are what it looks like when you ask a small model to be the thing that decides.

And if you have all four, you may not need the model either. That is worth checking first, because it was true for one of my three workflows and I did not expect it to be.

Where I was wrong

Twenty-two predictions, committed before I installed anything. Ten right, eleven wrong, one unresolvable.

Two of my seven confident calls broke. I was sure the 360M model would get most of the easy extraction cases, and it got 8 of 22 by failing at format rather than at finding. I was sure it would produce invalid output often, and it did so once in fifty, then emitted the same valid word forty-three times.

But the number that bothers me is not in the confident column.

Confidence	Right	Wrong
Confident	5	2
Leaning	2	7
Coin flip	3	2

My hunches did worse than my coin flips. Two of ten on the calls where I had a feeling but not much behind it, three of five on the ones I explicitly labeled as guesses.

That ordering is backwards and it is the most useful thing on the sheet. On the coin flips I knew I was guessing, so I guessed cleanly. On the leaning calls I was pattern-matching against intuitions about model behavior that turned out not to describe these models at all, and the feeling of having a reason was the problem. If I had marked all twelve of those as coin flips I would have scored better.

I will take that over a clean result. The honest version of this exercise is not "here are the numbers," it is "here is what I believed, here is what happened, and here is the gap."

The gap, this time, was mostly in the hunches.

The harness that produced these numbers had five measurement flaws in it, none of which manufactured an entire finding out of a missing directory and nearly got published. That is a separate post.

Everything here is reproducible: fixtures, runner, scorer, the committed predictions and the run record are at
github.com/kenwalger/right-size-the-model.
Fixture set v1. Read LIMITATIONS.md before citing any of these numbers:
two models, one board, one quantization, ninety-five cases. This is an illustration of a method, not a benchmark.

Data Types Across Programming Languages: What You Actually Need to Know

Ken W Alger — Tue, 06 Oct 2026 13:11:00 +0000

Every introduction to programming eventually reaches data types.

An integer is a whole number. A string contains text. A Boolean is true or false. A floating-point number represents numbers with fractional components.

Then you learn another programming language and discover that apparently none of this was quite as simple as advertised.

An integer in Python is not the same thing as an integer in C. JavaScript's number covers values other languages separate. Rust would rather make absence explicit than let null wander through a program. R has several different ways for something to be missing.

The definitions are not wrong. They are incomplete. A data type does more than tell a computer how to store a value. It expresses what a language assumes you should be able to represent, which distinctions matter, and which mistakes should be caught before the program runs.

So instead of memorizing another list of types, let's give several languages the same small collection of facts and see what happens.

Meet Alex.

Alex is about to have a complicated afternoon, mostly because of one missing middle name.

The deceptively simple data model

Suppose our program needs to represent a customer:

name = "Alex"
age = 42
discount = 0.15
middle_name = no value
favorite_numbers = 3, 7, 11
status = ACTIVE

Nothing exotic. Text, an integer, a decimal value, an absent value, a collection, and a status chosen from a known set.

If data types were merely different spellings for the same concepts, moving this model between languages would be syntax conversion.

It isn't. And watch which field causes the most trouble.

Python: let the values tell us what they are

Python makes our first version pleasantly straightforward.

name = "Alex"
age = 42
discount = 0.15
middle_name = None
favorite_numbers = [3, 7, 11]
status = "ACTIVE"

At runtime those values have types such as str, int, float, NoneType, and list. We did not declare them before assigning because Python is dynamically typed.

That makes Alex easy to get into the system. It also means constraints exist only if we deliberately add them.

Nothing about status = "ACTIVE" prevents a later status = "ACTVE". Python does not know our domain recognizes ACTIVE, SUSPENDED, and DELETED and has never heard of ACTVE. We can model that with an Enum, type hints, validation, or classes, but the bare string does not carry the rule.

Python's integers hold another surprise for anyone arriving from C or Java. A normal Python int is not limited to a fixed 32-bit range.

age = 999999999999999999999999999999999999

An absurd customer age, and a perfectly valid Python integer.

What nothing looks like in Python

Spelling	What it means	Who makes you deal with it
`None`	The only nothing available	Nobody

Already, "integer" has stopped meaning one universal thing. And None is a single, undifferentiated way of saying nothing is here, which will turn out to be a choice rather than an inevitability.

JavaScript: one `number`, and two different nothings

const name = "Alex";
const age = 42;
const discount = 0.15;
const middleName = null;
const favoriteNumbers = [3, 7, 11];
const status = "ACTIVE";

Similar enough, until we ask JavaScript about the values.

typeof age;         // "number"
typeof discount;    // "number"
typeof middleName;  // "object"

Yes, typeof null returns "object", a famous historical artifact that cannot be fixed without breaking existing code.

More important for our comparison, 42 and 0.15 share the ordinary type number, based on IEEE 754 double-precision floating point. JavaScript also has BigInt for integers beyond the safe range of number.

Absence is where it gets interesting. An uninitialized variable is undefined. A variable deliberately set to nothing is null. The language provides two spellings for emptiness and then declines to tell you which one means what. In practice one tends to mean "nobody has assigned this yet" and the other "somebody decided this is empty," except when a codebase uses them the other way around, or interchangeably.

Loose equality shows what the language really thinks. Zero, the empty string, false, and an empty array all agree they are the same thing. The two values that actually mean nothing is here do not join in:

0 == "";             // true
"" == false;         // true

null == undefined;   // true
null === undefined;  // false
null == 0;           // false
null == "";          // false

null and undefined are loosely equal to each other and to nothing else in the language. They form their own small island.

So JavaScript does distinguish absence from emptiness, and the rule is arguably the right one: a missing thing is not a zero thing. The problem is where that rule lives. It sits inside an equality operator famous for being unpredictable, next to a typeof that reports null as "object". Alex Dorey's JavaScript Equality Table renders every comparison as a grid, if you want the full picture and a certain amount of despair.

What nothing looks like in JavaScript

Spelling	What it means	Who makes you deal with it
`undefined`	Never assigned	Nobody
`null`	Deliberately empty, by convention	Nobody

A type system is already more than a vocabulary list.

C: representation becomes difficult to ignore

#include <stdint.h>

typedef enum {
    ACTIVE,
    SUSPENDED,
    DELETED
} Status;

typedef struct {
    char *name;
    int32_t age;
    double discount;
    char *middle_name;
    int favorite_numbers[3];
    Status status;
} Customer;

Representation is visible almost everywhere. We chose an explicitly 32-bit signed integer for age, double for the discount, an array of exactly three integers, pointers to characters for names, and an enumeration for status.

A null pointer might represent an absent middle name:

customer.middle_name = NULL;

But C does not attach our intended meaning to that pointer. NULL here could mean Alex has no middle name, that we have not loaded it yet, that the allocation failed, or that somebody forgot to initialize the struct. The program establishes the convention and the program is responsible for respecting it. The compiler will not help.

What nothing looks like in C

Spelling	What it means	Who makes you deal with it
`NULL` pointer	Whatever this program decided it means	Convention, and only convention

C also forces us to care directly about ranges and representation. A fixed-width signed integer cannot grow forever. This is not C being gratuitously difficult. C was designed around a different relationship between programmer and machine, and its type system reflects that.

Java: primitives, references, and the null question

enum Status {
    ACTIVE,
    SUSPENDED,
    DELETED
}

class Customer {
    String name;
    int age;
    double discount;
    String middleName;
    int[] favoriteNumbers;
    Status status;
}

Java distinguishes primitive types such as int, double, and boolean from reference types such as String, Customer, and arrays.

An int cannot be null. A String can.

That single asymmetry does a lot of work. It means absence is available for some of Alex's fields and structurally impossible for others, and the choice was made by whoever picked the field types rather than by anyone thinking about the domain. Unless we add conventions or tooling, String middleName does not tell us whether null is an expected state or a bug waiting for the right execution path.

What nothing looks like in Java

Spelling	What it means	Who makes you deal with it
`null`	An empty reference	Nobody
(not available)	Primitives cannot be absent at all	The compiler, by refusing

Java also has wrapper classes such as Integer, which means two things that sound like "an integer" behave differently:

int age;
Integer optionalAge;

One of those can be absent. The other cannot. Learning the keyword is only the beginning; the useful question is what guarantees come with it.

Rust: make some invalid states harder to represent

enum Status {
    Active,
    Suspended,
    Deleted,
}

struct Customer {
    name: String,
    age: u32,
    discount: f64,
    middle_name: Option<String>,
    favorite_numbers: Vec<i32>,
    status: Status,
}

Look at middle_name: Option<String>.

Instead of allowing a String to secretly contain null, Rust represents the possibility of absence in the type itself. The value is either Some(...) or None, and code consuming it has to deal with that possibility before it can get at the string.

This is the same information every other language has been carrying around implicitly, finally written down somewhere the compiler can read it. The other languages let you forget. Rust makes forgetting a compile error.

The Status enum similarly makes the valid alternatives explicit. We cannot accidentally assign ACTVE, because no such variant exists.

What nothing looks like in Rust

Spelling	What it means	Who makes you deal with it
`None`, inside `Option<T>`	This value may legitimately be absent	The compiler

A type system can encode valid states, not merely storage categories. That does not make incorrect programs impossible. It moves some classes of error from runtime behaviour into representations the compiler can reject.

The type system is participating in the design.

Go: useful zero values and deliberate simplicity

type Status string

const (
    Active    Status = "ACTIVE"
    Suspended Status = "SUSPENDED"
    Deleted   Status = "DELETED"
)

type Customer struct {
    Name            string
    Age             int
    Discount        float64
    MiddleName      *string
    FavoriteNumbers []int
    Status          Status
}

One important Go idea appears before we populate Alex at all: types have zero values. The zero value of an int is 0, of a bool is false, of a string is "". Pointers, slices, maps, functions, channels, and interfaces can be nil.

That creates a question. If Age contains 0, is Alex zero years old, or did nobody supply an age?

The type cannot answer. If the application uses a zero value to stand for “not supplied,” that state becomes indistinguishable from a legitimately supplied zero. If the distinction matters, the model has to represent it deliberately, which is why MiddleName above is a string rather than a string.
**What nothing looks like in Go*

Spelling	What it means	Who makes you deal with it
`0`, `""`, `false`	Zero/default value; cannot by itself tell us whether a domain value was supplied	The model must distinguish this if it matters
`nil`	Empty pointer, slice, map, channel, or interface	Nobody

Convenient defaults acquire meaning only when they enter a domain.

R: "missing" is not one simple state

name <- "Alex"
age <- 42L
discount <- 0.15
middle_name <- NA_character_
favorite_numbers <- c(3L, 7L, 11L)
status <- "ACTIVE"

R was designed around statistical computing, and its types and missing-value semantics reflect that.

NA represents a missing value, with typed forms such as NA_integer_ and NA_character_. R also has NULL, which is not another spelling of NA, and numeric computation can produce NaN.

These answer different questions. A missing observation in a dataset is not the same thing as the absence of an object. Neither is the same thing as a mathematically undefined result. R treats those as three distinct facts, because in statistics they are three distinct facts, and conflating them produces wrong answers rather than crashes.

Consider:

mean(c(10, 20, NA))
mean(c(10, 20, NA), na.rm = TRUE)

What nothing looks like in R

Spelling	What it means	Who makes you deal with it
`NA`	A missing observation	Functions ask, via `na.rm`
`NULL`	No object at all	Nobody
`NaN`	An undefined numeric result	Nobody

The second call makes an explicit methodological choice to exclude missing observations. The language made you say so.

The field that caused all the trouble

Alex's missing middle name has now been through seven general-purpose languages. Stack those seven tables on top of each other and the third column tells the whole story.

Python gave it None, one undifferentiated nothing. JavaScript offered two and left the meaning to you. C gave a null pointer whose meaning lives entirely in a convention the compiler cannot see. Java made absence available for reference types and impossible for primitives, as a side effect of a decision about memory. Rust put it in the type signature and refused to let you ignore it. Go made it indistinguishable from a valid value unless you reach for a pointer. R split it into three concepts because its users needed three.

Not one of those languages is wrong. They disagree about a prior question: how much does the programmer have to admit they do not know?

Representations of absence carry semantics whether or not the programmer acknowledges them. The difference between these languages is who is required to do the acknowledging, and when.

Which is why the missing middle name turned out to be more interesting than the integer.

The table gets weird quickly

Language	Typing	Ordinary integers	`42` and `0.15` same type?	How absence is spelled
Python	Dynamic	Arbitrary size	No	`None`
JavaScript	Dynamic	Fixed (`BigInt` for more)	Yes, both `number`	`null` and `undefined`
C	Static	Fixed width, chosen	Usually no	Null pointer, by convention
Java	Static	Fixed width	No	`null`, references only
Rust	Static	Fixed width	No	`Option<T>`, in the type
Go	Static	Fixed width	No	Zero values, `nil` for some types
R	Dynamic	Fixed width	Usually distinct	`NA`, `NULL`, `NaN`

Closed sets of alternatives vary just as much. Rust, Java, and C have real enums, Python offers Enum if you reach for it, Go builds them from a named type plus constants, JavaScript models them by hand, and R uses factors. Collections differ too: C, Java, Rust, and Go enforce an element type in the declaration, Python and JavaScript do not, and R's atomic vectors quietly coerce everything toward a common type.

There is no universal translation saying:

Python int = JavaScript number = C int = Java int = Rust i32 = Go int = R integer

Those types overlap. They do not carry identical guarantees, ranges, operations, failure modes, or assumptions.

The same name does not imply the same contract.

If you were designing a language, what would you choose?

This gets much more interesting when you reverse the problem.

Suppose you are designing a language.

You need an integer.

Easy.

Except now you have questions. Fixed widths, or one integer that grows until memory runs out? What happens on overflow? Should 3 become 3.0 on its own? Should "42" + 1 be legal, and if so, what should it mean?

Then absence, which by now you can see is the hard one. Do you have null? Can every reference be null, or do you require an explicit optional type? Do you distinguish a missing observation from a non-existent object? Whichever you choose, every program ever written in your language inherits that decision.

Then collections, where you decide whether elements must share a type and whether values can be moved or consumed. Then domain state, where you decide whether a programmer can say a value is only ever ACTIVE, SUSPENDED, or DELETED, and whether the compiler can prove every case was handled.

Finally, timing. When should a bad type relationship surface: while parsing, during compilation, at runtime, or only when the offending path finally executes?

There is no universally correct set of answers. There are consequences.

Every choice makes some programs easier to express, some mistakes harder to make, some implementations simpler, some runtimes more complicated, and some programmers wonder why on Earth the language designer did that.

Sometimes the answer is historical compatibility. Sometimes performance. Sometimes safety. Sometimes the problem domain.

And sometimes the language designer has a very good explanation and would appreciate it if everyone stopped bringing up typeof null.

One language where absence is not the question

Everything above is a general-purpose language, and every one of them had to decide what nothing means. That is not a universal preoccupation. It is what happens when a language expects you to model a world.

The e language, commonly associated with Cadence Specman, was designed for hardware verification. Verification code describes legal ranges of values, generates constrained test data, and explores systems whose state spaces are far too large to enumerate by hand. So its declarations carry constraints rather than storage categories:

struct customer {
    age : uint (bits: 8);
    keep age in [18..120];

    status : [ACTIVE, SUSPENDED, DELETED];
    keep soft status == ACTIVE;
};

There is no absence card for e in this article, and that is the point. In ordinary application programming we ask what type of value this is. A verification environment asks what values are legal here, and how valid combinations should be generated so we can explore what the system does.

If you look only at Python, JavaScript, Java, and C, it is easy to assume programming languages are all solving approximately the same problem with different punctuation.

They aren't.

Types are compressed language philosophy

At the beginner level, "a type tells us what kind of data a value contains" is perfectly serviceable. Eventually that runs out of road. A type can tell us what values exist, how they are stored, which operations are permitted, which conversions happen silently, what counts as absence, whether absence must be acknowledged, whether alternatives form a closed set, and in some languages how long a value lives and who owns it. "Integer, string, Boolean, float" is not the definition of a type system. It is the introductory vocabulary.

Which is why learning the types in a new language beats memorizing a conversion table.

Python's arbitrary-precision integers are a tradeoff. So is C asking you to care about width. Java's primitive and reference split quietly decides which of your fields are allowed to be empty. Go buys simplicity with zero values and pays for it with ambiguity at the domain boundary. R distinguishes forms of missingness because missing observations are a normal feature of statistical work, so its problem domain is visible in its types. A verification language making constrained generation first-class tells you what its designers expected programmers to spend their days doing.

A language's types are partly a record of what its designers thought mattered, and partly a record of what they were willing to let you avoid thinking about.

They are compressed language philosophy.

Alex survived

After all of this, Alex is still 42, still has no recorded middle name, still likes 3, 7, and 11, and remains active. Nothing about the underlying person changed. But every language forced different decisions about how those facts became computation, and the fact that changed most was the one that was not there.

Knowing that Python has int, Java has int, and Rust has i32 is useful when you are trying to get a program running. Understanding why they are not interchangeable is useful when you are trying to understand languages.

Once you start asking what a language lets you represent, what it makes you acknowledge, and what it refuses to let you say, data types stop being the boring chapter before loops. They become one of the quickest ways to see what a language believes programming should be.

What is the strangest, most useful, or most frustrating type decision you've encountered in a programming language? More importantly, what problem do you think the language designer was trying to solve with it? Or, if you were designing your own language, which of these decisions would you make differently?

Enterprise MCP Gateway to Govern MCP Tool Access

Ken W Alger — Tue, 06 Oct 2026 04:57:00 +0000

Model access controls what an AI system can know and generate. Tool access controls what it can do.

The previous posts in The AI Control Plane have moved progressively closer to the point where policy becomes action. Governance established who gets to decide. Virtual keys turned those decisions into runtime policy. RBAC and access profiles constrained who could change that policy. User provisioning made those permissions follow people as they joined, moved, and left.

Bifrost is the open-source AI gateway through which this series has been examining those control-plane concerns, with its implementation available in the Bifrost GitHub repository.

MCP tool access changes the stakes. The Model Context Protocol defines tools as capabilities exposed by servers that clients can discover and invoke. A model allowed to answer a question can produce a bad answer. A model allowed to invoke a tool can change another system. It can create a ticket, query a customer database, update a record, trigger a deployment, send a message, or call whatever capability an MCP server exposes.

The question is no longer only which model may receive a request. It is which actions that model can reach, under whose authority, and what happens when a request crosses from inference into execution.

Why Is MCP Tool Access Different From Model Access?

Model governance controls destinations and consumption. Which providers are approved, which models a workload may use, how much it may spend, and what rate limits apply. Tool governance adds capability.

Consider a customer support agent with access to three tools: lookup_customer, issue_refund, and close_account. All three may live on the same MCP server, behind the same authentication, exposed through the same connection. They do not have the same consequences.

That is why the useful unit of governance is not the server. A server is a deployment boundary. A tool is an action boundary, and the two rarely line up.

Authentication does not close the gap either. Valid credentials for an MCP server prove a connection can be made; they say nothing about whether a given workload should reach every capability behind it. This is the same separation the virtual key post drew between holding a provider credential and being permitted to use a particular model, and it gets sharper when MCP authentication is per-user. A person may legitimately authenticate to an external service while the AI workload acting on their behalf should still receive a subset of what that service offers. The credential establishes identity at the external boundary. The control plane still has to govern capability at the tool boundary.

What Should MCP Authorization Policy Control?

The simplest policy is an allow list, and it does more than it appears to, because filtering constrains discovery as well as execution. A workload never needs to be shown capabilities it will not be permitted to invoke. Bifrost applies that filtering per request, per client, or per virtual key, which means tool access can attach to the same policy boundary that already governs provider access, model allow lists, budgets, and expiry.

Enterprise authorization needs more structure than a growing collection of individual tool grants. Access profiles provide a reusable policy layer for grouping permissions around a role or workload rather than configuring each identity independently. A support workload, for example, can inherit the provider, model, and MCP permissions appropriate to support without requiring those decisions to be recreated every time another user or key is provisioned.

Virtual MCP servers provide another boundary by controlling the capability surface presented to a workload. Rather than treating every tool exposed by every connected MCP server as one flat namespace, an organization can present a curated MCP surface appropriate to a particular application or class of users. Tool filtering can then narrow that surface further to the specific actions a workload is authorized to discover and invoke.

Together, those layers answer different authorization questions. An access profile establishes the policy associated with an identity or workload. A virtual MCP server establishes the MCP capability surface available to it. Tool-level policy determines which actions within that surface are actually reachable.

That distinction matters because enterprise policy gets nuanced quickly. A research agent may search documentation and retrieve customer information without modifying either. A support agent may open a case while refunds route through a separate path. An engineering agent may inspect deployment state while production mutation stays out of reach. Policy should track the consequence of the action rather than the convenience of the integration.

Because those permissions live in the control plane rather than in twenty repositories, changing the decision does not require twenty releases. It also means refusal happens in the right place. If a workload is not permitted to invoke issue_refund, the useful outcome is not that the refund service rejects the call later. It is that the refund service never sees it. The application receives a deterministic policy refusal instead of an unpredictable downstream error, and the external system is never asked to adjudicate an unauthorized attempt.

What Changes When an Agent Can Execute Tools Automatically?

There is a meaningful difference between a model suggesting a tool call and infrastructure executing one, and Bifrost's default preserves it. Tool calls returned by a model are suggestions, not actions:

1. POST /v1/chat/completions  
   → model returns tool call suggestions, not executed

2. Application reviews the proposed calls  
   → apply business rules, request human approval

3. POST /v1/mcp/tool/execute  
   → execute approved calls explicitly

4. POST /v1/chat/completions  
   → continue the conversation with tool results

Step two is the pause. It is where an application can inspect a proposal, ask a human, apply additional logic, or refuse outright.

Agent mode removes it. Tools named in the tools_to_auto_execute configuration run without waiting for approval, which is useful precisely because it eliminates friction and consequential for the same reason.

So the authorization decision moves earlier. If no human will be present when the action is taken, the control plane has to establish beforehand which actions are acceptable. An allow list stops being a convenience for hiding irrelevant tools and becomes part of the safety boundary around autonomous execution. That does not mean every call needs an approval dialog. It means the absence of a human at execution time makes pre-established policy more important, not less.

How Should Enterprises Think About MCP Tool Blast Radius?

A workable way to classify tools is by what happens when the authorization turns out to be wrong.

Capability	Example	Primary Consequence	Governance Posture
Read	Search documentation	Information exposure	Allow by role and data scope
Create	Open support ticket	New system state	Restrict destination and ownership
Modify	Update customer record	Existing state changes	Narrow scope and retain evidence
Financial	Issue refund	Monetary loss	Explicit grant and tighter limits
Destructive	Delete account	Irreversible or costly	Exceptional access or added approval

The categories matter less than the exercise. Two tools on the same server can sit three rows apart, which is the concrete reason server-level grants are too coarse.

Why Do Individually Safe Tools Become Unsafe Together?

Here is the part that no allow list solves.

A workload permitted to read customer data and to send email has a different effective capability from one permitted to do either alone. An agent holding a database tool, a payment tool, and outbound messaging is not carrying three permissions. It is carrying a workflow, and nobody approved the workflow. Each grant was reviewed on its own and each was defensible on its own.

This is an uncomfortable property of agentic systems: capability emerges from combinations of individually reasonable permissions, and the emergent capability is a property of the set rather than of any member of it. Filtering evaluates tools one at a time, because that is what filtering is. Reviewing grants one at a time will not surface it either, for the same reason.

There is no product answer as no gateway-level enforcement exists today. What there is, for now, is a review practice: when adding a tool to an existing grant, ask what the new tool makes possible in combination with what is already there, rather than whether the new tool is acceptable by itself. That is a weaker control than enforcement, and it is worth naming as a gap rather than papering over it.

A second assumption runs underneath everything above: that the MCP server itself is trustworthy, and the only open question is which of its tools a given workload should reach. That assumption deserves its own scrutiny, because a compromised or careless server can return content engineered to be read as instruction rather than as data. I have written separately about classifying MCP servers by trust zone, which is the layer beneath the one this post governs.

Who Authorized This Tool, and What Happens When They Leave?

Filtering answers whether a call is permitted right now. It does not answer why.

"The developer added the tool to the agent" describes implementation, not authority. The permission may have arrived through a team role, an access profile default, a temporary exception, or a direct assignment created during an incident at two in the morning. Those paths have different lifecycle behavior, which is where the provisioning problem returns. A tool grant attached to an identity has to disappear when the authority behind it disappears. Deleting a user or key causes Bifrost to clean up the associated per-user credentials. Session-keyed credentials and upstream provider tokens, however, may have separate revocation lifecycles that administrators need to account for.

And a tool grant attached to a workload with no human identity needs an owner and a lifecycle of its own, because nothing will trigger one on its behalf.

MCP governance is not a list of allowed tool names. It is the connection between capability and authority.

Why MCP Governance Belongs in the AI Control Plane

This series has worked through increasingly specific forms of control: organizational governance, workload policy, administrative authority, identity lifecycle, and now capability. Tools are where those controls finally meet something able to change the world outside the AI stack.

Which is why tool governance cannot stop at authentication, server access, or whether a model supports function calling. Enterprises have to decide which capabilities are visible, which are executable, which demand stronger authority, and how all of that changes when the people and workloads behind them change. Bifrost brings those decisions into the same control plane as model access, routing, and budgets, and because it is open source, the MCP implementation can be inspected rather than taken on trust.

The principle outlives any implementation. Model access governs what AI may ask. Tool access governs what AI may do.

Five layers in, the remaining question is not which controls exist. It is what order to build them in, and what breaks when you get the order wrong.

Bifrost brings these controls into the AI control plane, giving teams a central place to manage MCP authorization alongside the rest of their AI infrastructure. To see how that works in practice, book a Bifrost demo.

This article was commissioned by the Bifrost team. Links to kenwalger.com and to the Sovereign Systems Specification point to my own work. The architectural perspective and conclusions expressed here are my own.

Views Measure Views

Ken W Alger — Thu, 01 Oct 2026 19:32:00 +0000

Nine years in, I finally worked out what else to count.

A writer I follow, Sylwia Laskowska, recently published a post about accidentally becoming a blogger. She has been writing on DEV for about a year, and the numbers attached to that year are impressive: hundreds of thousands of views and tens of thousands of followers.

I have been on DEV for more than nine years. This morning I am at 48,408 total views.

Before I go any further, I want to be clear about something, because the essay that usually follows a comparison like that is insufferable. Building a large readership for accessible, useful developer writing is genuinely difficult, and doing it in a year is more difficult still. I would like more people to read my work. I can learn a great deal from writers who are better than I am at audience building, topic selection, accessibility, and community participation.

So this is not a piece about why small numbers are secretly good. It is a piece about what I spent nine years failing to separate.

The number that stopped me using one of my metrics

I recently pulled nine years of my own data out of the DEV API, mostly to make a chart of follower growth with my publication dates marked on it, so I could see which posts moved the line.

The chart was useless, and the reason is instructive.

In 2026 I gained roughly eighteen thousand followers. My 2026 posts have about fourteen thousand views between them. You cannot acquire eighteen thousand readers from fourteen thousand page loads. The daily follow rate sits around 130 and does not respond to whether I publish anything, and about 37% of the usernames carry auto-generated hex or numeric tails. It is reciprocal-follow farming, it is endemic, and it has nothing to do with me or my writing.

Here is the cleanest version of it. Since the middle of September, my follower count has gone from 18,148 to 21,048. Over the same fifteen days, my total view count went from 46,774 to 48,408.

Two thousand nine hundred new followers. One thousand six hundred and thirty-four new views.

I gained nearly twice as many followers as readers, and a follower is supposed to be a reader who liked something enough to want more. That number had been sitting on my profile for months looking like evidence of something.

A page view is at least an honest measurement. Somebody loaded the page. That is real information, and I think "vanity metric" is an unfair label if it is taken to mean meaningless.

The trouble starts when we quietly change the claim from "this post received more views" to "this post was more successful."

Successful at what?

A page view is real. It just is not the whole story.

In corporate content the answer to that question is usually explicit. A post might exist to attract someone searching for a problem, introduce a product, move that reader toward a trial, and eventually help create a customer. Ten thousand views with no downstream behaviour may be worth less to that company than five hundred views that put twenty qualified developers into the funnel.

Personal writing has a funnel too, just a much vaguer one. Someone reads one article, encounters another a month later, starts recognising the name, follows, leaves a substantive comment, references the work elsewhere. The page view was real. It was not necessarily the outcome.

It also helps to remember that a broadly useful JavaScript tutorial and an article about authority boundaries in AI-generated software are not competing for the same reader. One is relevant to a large share of a developer community. The other starts with a much smaller pool of people who already care about capability security and the distinction between correctness and authority.

That is not so different from comparing the audience for a popular fantasy novel with the audience for a presidential memoir. Both books can be excellent. Both can do exactly what their authors intended. Their potential readerships are still radically different.

Audience size is partly a property of the artifact and partly a property of the market around it. That sounds obvious about books. Writers forget it remarkably fast while staring at a dashboard.

An article can also have a desired consequence without a button under it. Sometimes I want someone to challenge the argument. Sometimes I want a developer to recognise a problem in their own system. Sometimes the entire intended outcome is that a reader leaves with a question they were not asking ten minutes earlier. That still gives me something to evaluate against. It just refuses to appear as conversion=true in an analytics dashboard.

Quality, audience fit, and distribution are different problems

I have come to think of online writing as at least three separate problems.

Writing quality is the craft problem. Is the argument coherent? Is the explanation useful? Did I do the research? Is there something here worth another person's time?

Audience fit is the relevance problem. How many people where I publish are likely to care about this subject? How much prerequisite knowledge does it demand? Can someone scrolling a feed see immediately why the question matters to them?

Distribution is the discovery problem. How does the article reach those people? Search, followers, newsletters, speaking, community participation, platform curation, links from other writers, or some combination.

Those interact, but they are not interchangeable. An article has to clear all three, and failing any one of them produces the same disappointing number for completely different reasons.

flowchart LR
    W["Article"] --> Q{"Quality"}
    Q -->|"weak"| X1["Nobody finishes it"]
    Q -->|"strong"| F{"Audience fit"}
    F -->|"wrong room"| X2["Few people care"]
    F -->|"right room"| D{"Distribution"}
    D -->|"not found"| X3["Nobody sees it"]
    D -->|"found"| R["Readers"]

    classDef gate fill:#FFFFFF,stroke:#166534,color:#14532D,stroke-width:2px;
    classDef miss fill:#FEF2F2,stroke:#991B1B,color:#7F1D1D,stroke-width:2px;
    classDef good fill:#E8F3EE,stroke:#166534,color:#14532D,stroke-width:2px;

    class Q,F,D gate;
    class X1,X2,X3 miss;
    class W,R good;

That is the diagnostic value of separating them. Three posts can land at 80 views apiece and need three entirely different responses. A good article can have poor audience fit. An accessible article can have excellent fit and no distribution. A technically modest piece can answer a question a hundred thousand people are asking today. A strong argument can address a question five hundred people know they have.

For most of my writing life I concentrated almost entirely on the first variable and assumed distribution would sort itself out. Sometimes it did. Frequently it did not. My own numbers say that plainly: about 4% of my traffic comes from search, which for someone whose best-performing historical work is evergreen reference material is a distribution problem rather than a quality one.

I did not start with a content strategy

Nine years ago my public technical writing grew out of databases, because databases were the work I was doing and the community I was in. I did not sit down with a personal-brand document. I wrote about what I knew, what I was learning, and what developers were asking about.

Those articles accumulated into something larger. People began associating my name with certain subjects, questions led to more articles, and some of that work became durable enough to keep attracting readers years later. My database writing eventually included MongoDB's Building with Patterns, one of the most heavily visited bodies of content I worked on there.

That history matters when I wander. If I publish one Rust article today, it does not arrive with nine years of association between my name and Rust behind it. That does not mean developers are uninterested in Rust, or that my readers dislike it. It may simply mean I have not given a Rust audience any reason to know who I am.

Those explanations imply completely different responses. If I wanted Rust to become a real part of my writing, one underperforming article would tell me almost nothing. I would need several useful pieces, participation in that community, and time. If I do not want that, the article stays an interesting experiment.

"This article performed poorly" is an observation. "There is no audience for me here" is an interpretation.

Personal writing gets to discover its strategy

I have spent enough of my career around corporate developer content to know that content strategy matters. A company generally knows why it is publishing: which developers it wants to reach, which capabilities it needs explained, which search terms it wants to own. The strategy should exist before anyone fills the editorial calendar.

Personal writing is stranger. You can chase a question because it bothered you on Tuesday. You can abandon a series when you have nothing else useful to say. You can spend weeks on a technical experiment and then publish something ridiculous because you started wondering whether all the photographs on your phone technically make it heavier.

You can also discover the strategy after you have written enough to see the pattern. That has increasingly been my experience. Articles I thought were about AI verification, provenance, missing information, audit evidence, memory, and authority turned out to be different views of the same few questions. I did not design that body of work and then manufacture articles to fill it. The writing is how I found it.

The conversation that prompted this piece made me realise that is less different from corporate strategy than I assumed. Sylwia described writing mostly by intuition while still making choices about what she wants to be known for, which audiences interest her, which adjacent topics fit, and which opportunities she ignores.

That is a content strategy. It just has a governance structure of one.

A company may need content, DevRel, product marketing, SEO, and leadership involved in deciding whether a newly discovered audience matters. A personal writer can notice something in the comments on Tuesday and run the experiment on Thursday. The strategic question is nearly identical. The path from observation to decision is not.

Signals are not instructions

This matters because audiences talk back. A recurring question in the comments may reveal an adjacent audience you did not know you had. Search traffic may show people finding an article for a reason you never anticipated. A series may attract platform engineers when you thought you were writing for application developers.

That is useful information. It is not an order.

A writer can discover that beginner tutorials have an enormous reachable audience and still decide not to build a body of work around them. A company can discover that a group of users loves a product for an unexpected use case and still decide that market does not fit the strategy.

Discovering an audience is not the same as deciding to serve it.

This is where "write more of whatever did best last week" collapses. It is the same loop with one step deleted.

flowchart LR
    A["Write"] --> B["Publish"]
    B --> C["Signals<br/>views · comments<br/>search terms · who shows up"]
    C --> D{"Is this where<br/>I want to go?"}
    D -->|"yes"| E["Build an audience here"]
    D -->|"no"| F["Note it. Leave it alone."]
    E --> G["Strategy"]
    F --> G
    G --> A

    classDef step fill:#E8F3EE,stroke:#166534,color:#14532D,stroke-width:2px;
    classDef signal fill:#FFFFFF,stroke:#5CA08A,color:#14532D,stroke-width:2px;
    classDef gate fill:#FEF2F2,stroke:#991B1B,color:#7F1D1D,stroke-width:2px;

    class A,B,E,F,G step;
    class C signal;
    class D gate;

The diamond is the part that gets skipped. Without it the loop still runs, it just runs on autopilot, and the writer ends up somewhere chosen by whatever the feed rewarded in a given week.

Metrics tell you what happened. Readers reveal opportunities you did not know to look for. Neither one gets to decide what you want the work to become. The feedback loop needs interpretation.

I eventually wrote down some rules

Once I could see a body of work forming, it became tempting to turn every passing thought into another strategic article. So I wrote myself a gate.

For the deliberate part of my technical writing, I now ask whether there is a disputable claim, whether investigating it will put pressure on an actual artifact, whether I have standing through a project or experiment, whether it advances rather than repeats the larger body of work, and whether a reader should do or question something differently afterward. I also want to know what would falsify the claim before I start assembling evidence for it.

The artifact-pressure test is deliberately hard. If investigating an idea will not change code, a specification, an ADR, a schema, or a demo, it probably does not belong in that stream. The investigation also has to be capable of failing. Building fixtures that encode what I already believe proves very little.

That produces a shape I have become fond of:

Here is what I thought.
Here is what would have convinced me I was wrong.
Here is what I built.
Here is what happened.
Here is what changed.

Those are not my rules for everything. I deliberately keep another lane with almost no gate at all: career observations, language experiments, satire, community responses, project archaeology, and pure curiosity only need to be worth writing.

A publishing strategy should help me recognise strong work. It should not make me ask permission before being curious.

Some of it is luck

There is one variable I cannot put into a strategy with any confidence.

I can study years of data and conclude that Tuesday at 9:00 a.m. Pacific is the right time to publish. That says nothing about whether it is right for this article. A major news event may take the morning. Three other posts aimed at the same readers may appear within the hour. A moderator may promote something. A Gem may land. Another writer with a large audience may link to you. None of that says anything new about the quality of the work, and each can transform its distribution.

Strategy does not eliminate luck. It changes the conditions under which luck operates. You can improve the writing, understand the audience, participate in the community, publish consistently enough that people know you exist, and make the work easy to find.

You can engineer more opportunities for an outcome without engineering the outcome itself.

And when luck does hand you an unexpected success, strategy comes back. A surprising audience appearing is another signal, not a mandate. You still have to decide whether it points somewhere you want to go.

What I actually track now

I still look at views. Pretending otherwise would be silly. If one article gets 5,000 and another gets 40, I want to know why. I just no longer think the first was 125 times more successful.

The outcomes I care most about do not fit in a platform dashboard. A reader challenges a claim and I change the model. A comment exposes a missing invariant. An experiment breaks the answer I expected to publish. An article changes code, a specification, or an architecture decision.

That has become much less theoretical lately.

I published an article that began with a refractometer and a batch of homemade wine, arguing about the difference between a measurement and the state we infer from it. The comments pushed it considerably further: version the correction rule, spend more measurement budget on consequential baselines, decide how conflicting instruments get adjudicated before seeing the readings, distinguish evidence that survives a restart from evidence that dies with the process.

An article about authority boundaries in AI-generated code did the same thing. Readers pushed on capability lifetime, consumable authority, semantic authority diffs, denied-call telemetry, and who is permitted to modify the authority boundary itself.

Those comments did not just increase engagement. They changed the model. Those are propagation effects rather than distribution metrics, and I have started tracking them separately: research-induced change, engagement from people with standing outside my field, independent use of a concept, substantive challenges and extensions, and whether writing has started consuming so much time that the projects supplying it have stopped moving.

My numbers support the split more cleanly than I expected. Across nine years I have 1,372 reactions and 683 comments. Roughly one comment for every two reactions is a strange ratio, and it is concentrated almost entirely in recent work. My 2017 tutorials pulled more than twice the traffic of everything I wrote in 2026 and produced 26 comments in three years. The 2026 essays, with a third of the traffic, have produced hundreds. One of them has a comment thread 35 replies deep.

One body of work got found and skimmed. The other gets read and argued with. For years I evaluated both with the same number.

A related consequence: a technically unsuccessful investigation can make a successful article. If I start with a hypothesis, state what would falsify it, build something capable of producing an answer I do not control, and find out my model was wrong, that is useful. Possibly more useful. It tells the reader something, it changes the artifact, and it often exposes a better question than the one I started with.

I have a line in my current content plan that says missing a publishing day beats manufacturing a weak post. Nine years ago I am not sure I would have been comfortable with that.

Nine years later

I am still working this out. I still publish things and wonder whether anyone will care. I still occasionally write something I expect to perform well and watch it vanish. I still publish something on a whim and find it landed exactly where it needed to.

The difference is that I now have more ways to recognise success when it shows up wearing something other than a large number.

A successful post might reach 30,000 people. It might produce a conversation that changes the next article. It might expose a flaw in an architecture. It might give someone language for a problem they were already having. It might lead to code. It might turn out to be part of a body of work whose shape I could not see when I wrote the first piece.

And sometimes it might simply be an article I wanted to write, written well enough that I am still happy to have my name on it years later.

Views measure views. They are a real measurement of a real thing, and they do not independently measure rigour, usefulness, influence, audience fit, changed behaviour, changed artifacts, or whether the writing moved me toward a better question. Sometimes those correlate. Sometimes they do not. Mine told me almost nothing for nine years, and the twenty-one thousand followers told me less.

The useful question is not "Was this post successful?"

It is "What did I want this post to do, and what happened because I published it?"

Those are much harder numbers to put on a dashboard. I think they are also the ones worth learning to notice.

Thanks to Sylwia Laskowska for the conversation that prompted this, and for the encouragement to write some of it down.

The Most Useful Line on Your AI Cost Report Is the One You Can't Explain

Ken W Alger — Thu, 01 Oct 2026 13:16:00 +0000

Attribution, allocation, and why "unknown" belongs in the schema.

This piece grew out of a comment thread on Sarvar Nadaf's Per-Agent Cost Tracking for Multi-Agent AI on AWS. The schema below was worked out in that conversation, in public, and it is better for it. Where a specific idea came from the exchange, I have tried to say so.

Most AI cost dashboards answer one question well: how much did this run cost. Total tokens, model spend, per-agent spend, latency, tool usage. Those are real and useful numbers, and for a while they are enough.

They stop being enough the moment your system becomes a composition. Once a request flows through a retriever, a knowledge graph, three specialist agents, and a supervisor that synthesizes their output, the total tells you almost nothing about what to change. A run can be correct, return HTTP 200, look healthy in every latency-and-errors panel, and still cost forty percent more than an identical run that produced the same answer. The overspend is real. It is just not anywhere you are looking.

To find it, you have to stop asking where the money was spent and start asking what caused it to be spent. Those are different questions, and the gap between them is the whole subject of this piece.

Where a Cost Is Incurred Is Not What Caused It

Consider a retrieval operation that costs $0.004: searching, ranking, fetching. That is the direct cost, and it is easy to attribute. It happened on that span, you can measure it, done.

Now suppose that retrieval returned 20,000 tokens, and all of them were hydrated into a supervisor's context on the next step. The supervisor then costs $0.009. How much did the retrieval really cost?

The direct answer is still $0.004. But that is no longer the interesting answer, because the retrieval also caused cost somewhere else. It inflated the supervisor's context, and some portion of that $0.009 exists only because the retrieval handed it too much material. The cost was incurred at the supervisor. It was caused, in part, at the retriever.

This gives you two distinct dimensions, and a useful cost model has to carry both:

Where the cost was incurred. This is just the span. Directly observed, low ambiguity.
What caused or contributed to it. This is the interesting axis, and it is the one no aggregate dashboard shows.

A record that captures both might look like this:

span_id: retrieval-104
direct_cost: $0.0040
primary: retrieval
returned_tokens: 20000

downstream:
  span_id: supervisor-105
  attributed_cost: $0.0021
  caused_by: retrieval-104
  attribution_method: proportional

The dollars are single-counted. We do not charge the $0.0021 twice. The supervisor genuinely incurred it; the retrieval genuinely contributed to causing it; and the record says both without inventing money. What we have added is lineage: a link from a downstream cost back to the decision that helped produce it.

The Hard Part Is Honesty About How You Know

Here is the question that breaks naive versions of this: how do we know retrieval-104 actually caused $0.0021 of the supervisor's cost, and not some other amount?

Sometimes you can measure it. If you have a controlled comparison where the only meaningful change is that retrieval result, the delta is real evidence. Supervisor costs $0.006 without the retrieved material and $0.009 with it, so roughly $0.003 of downstream cost is attributable to that retrieval. That is measured causation, and it is the strongest claim you can make.

Most production traces do not give you that. In a real run the supervisor is carrying system instructions, conversation state, the outputs of other agents, tool results, and the retrieved material, all at once. There is no clean counterfactual. So you fall back on allocation: split the supervisor's context cost proportionally by the tokens each source contributed. That is a reasonable method. It is not measurement, and the receipt must not pretend it is.

This is why the single most important field in the whole schema is not a dollar amount. It is this:

attribution_method:
  - measured_delta
  - proportional
  - estimated
  - unknown

That field is what keeps the entire model honest. It stops a proportional guess from masquerading as measured causation. With it, a line can say "retrieval span 104 contributed an estimated $0.0021 of downstream context cost, allocated proportionally by hydrated token share," and every word in that sentence is defensible, because the method is stated. Without it, the same $0.0021 acquires a precision the evidence never earned.

Resist collapsing this into a confidence score. A number like confidence: 0.82 feels rigorous and gives you nothing, because now you have a second number whose provenance you have to go investigate. measured_delta, proportional, estimated, and unknown each tell you why you are entitled to believe the figure. The method is the provenance. A score would hide it.

Show the Method Where the Decision Is Made

A natural instinct is to keep the attribution method as drill-down metadata, out of the main view, so the report stays clean. That instinct is wrong, and it is wrong for the same reason aggregate dashboards are wrong: it makes two different claims look equivalent.

The method belongs inline, next to any attributed cost, with one sensible exception. A directly observed cost carries no ambiguity and needs no method tag:

retrieval-104   RETRIEVAL   $0.0040

There is nothing to disclose there; it was measured on the span. But the moment a number is attributed rather than observed, the method has to ride along:

retrieval-104 -> downstream CONTEXT   $0.0021   proportional

Drop the word proportional and that $0.0021 visually becomes as solid as the $0.0040 above it, which is a lie of formatting. A report that hides the distinction between what it measured and what it allocated has committed the same sin as the dashboard that only shows a total. If two numbers make materially different claims, the interface must not make them look the same.

So the main report shows amount, category, direct versus downstream, and method. The drill-down holds the evidence behind the method: hydrated token counts, comparison runs, parent-child span references, the assumptions the allocation rests on. The decision surface stays readable; the receipt is one click away, not dumped into the table.

"Unknown" Is Not a Gap in the Accounting

Every honest version of this schema has to allow unknown as a real value, not a placeholder you feel bad about. And once you sit with it, the unknown rows turn out to be the most useful rows in the report.

A high downstream cost with unknown attribution is not incomplete bookkeeping. It is the system telling you exactly where your observability boundary stops letting you explain its own behavior. It is pointing at the place where you cannot yet answer "what caused this," which is the place most worth instrumenting next. A tidy report with no unknown rows has usually not achieved understanding. It has hidden its ignorance behind confident allocation.

There is also a real reason unknown is sometimes the only honest answer: context is not additive. An extra 5,000 tokens of context does not simply add a proportional slice of cost. It can change caching behavior, alter the reasoning path the model takes, or change how much output the model generates downstream. When that happens, token share and cost share stop mapping to each other cleanly, and any proportional number you report is a polite fiction. In those cases the schema should say unknown and mean it, rather than allocate a figure it cannot defend.

Read that way, the report stops being a statement of where you paid and becomes a map of two things at once: what you can explain about your spending, and where your ability to explain it runs out. The second map is the one that tells you what to build.

What This Actually Costs to Build

The reason this is not a research project is that most of the structure already exists. If your traces are parent-child spans, the causal lineage is physically present already; a downstream cost sits under the decision that produced it. You are not inventing a new tracing mechanism. You are making the attribution semantics explicit on top of a trace you already record.

Concretely, that is a small number of additions. Stamp a primary category on each span. Add a contributes_to link and a cause on spans that produce downstream effects. Add the attribution_method on any attributed cost. Then roll the report up along two axes, category and direct-versus-downstream, and let unknown be a first-class row rather than a swept-under one. The intelligence is not in the plumbing. It is in having the honest method field and being willing to publish the unknown rows.

One warning from the same conversation that produced all this: how you record and how you attribute are coupled. Change the way spans are emitted and you can silently break the logic that reads them. The defense is the same one that makes the whole model trustworthy, a known-good baseline you compare against, so that when your instrumentation shifts under you, the numbers move and you notice.

The Point

We have spent a lot of effort making AI spend visible. Better token counts, per-agent breakdowns, nested traces. All of it answers "how much." Almost none of it answers "why," and "why" is the only version of the question you can act on.

The move from one to the other is not a bigger dashboard. It is a small, honest schema: separate where a cost was incurred from what caused it, state the method behind every attributed number, and treat the costs you cannot explain as signal rather than embarrassment. Do that, and the report stops telling you what you spent and starts telling you what to fix, including, in the unknown rows, where to look first.

Even the cost report, it turns out, needs provenance. Not just the amount and the category, but how sure you are about who to blame. That last column may be the most useful one on the page.

With thanks to Sarvar Nadaf, whose post and the conversation under it produced this schema, and to the commenters in that thread who pushed on the baseline and the propagation. The receipt is better for the argument.

A wine cellar that remembers what you used to believe

Ken W Alger — Wed, 30 Sep 2026 22:49:21 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange

What I Built

Cellar is a wine collection that remembers what you used to believe.

Most cellar apps answer one question: what do I have right now? Cellar answers a harder one. What did my cellar look like on 1 June 1999? What did I think that bottle's drinking window was in 2018, before a later tasting changed my mind? Which bottles were at peak last year while I wasn't paying attention, and are now past saving?

Those questions rule out the obvious data model. A status field on a bottle tells you what is true today and nothing about 2004. A drinkUntil field on a wine tells you what someone currently believes and destroys what they believed before.

So Cellar stores neither. It stores events and claims:

Acquisitions and consumptions are documents with dates, not fields on a bottle. A bottle is consumed as of date T if a consumption event exists on or before T.
Assessments are dated, attributed claims about when a wine should be drunk. Nothing overwrites a window. The current window is resolved from the claims that existed at T, by explicit rules: my own tasting note outranks the producer's, which outranks a critic's, and within a tier the most recent claim wins.

Everything else follows. Move a date control backward and acquisitions disappear, consumptions reverse, and drinking windows change as the controlling claim changes. Move it forward and unopened bottles age through their windows.

The design rule underneath: do not store what is true now when you may later need to know what was true then.

There are three surfaces, with different jobs:

Sanity Studio authors and governs the ledger. Six document types, a custom review queue, two document actions, and a read-only workflow field.
A Sanity App, built on the App SDK and running inside Sanity, interprets it. Three views, all computed live from the event log by a pure TypeScript package that knows nothing about Sanity.
A public page is that same App without an account. Read-only by construction rather than by convention, since there is no token in the bundle and the dataset's public ACL grants reads only.

What it deliberately does not have is a second CRUD interface. Studio already is one, so adding bottles and recording consumptions happens there. The App interprets; it does not manage.

Who it is for, honestly: me. But the argument generalizes past wine, which is the reason I built it.

Demo

Open it with no account: kenwalger.github.io/Cellar

Inside Sanity (organization members): The Cellar app

The same 542-bottle ledger, read at three moments:

Date	In cellar	Drinking	Past window	Not yet acquired
1 June 1999	4	4	0	536
23 September 2026	248	166	33	0
23 July 2035	248	62	182	0

In 1999 the whole cellar is four bottles of 1993 Château Mouton Rothschild, with two already opened. By 2026 it is 248 bottles. Projected to 2035, those same 248 bottles are still there, and 182 of them have gone past window.

The cellar never loses bottles. It loses opportunities.

Past:

Present:

Future:

Drink Soon lists what closes within twelve months, and every row names the claim that decided it: which tier, whose, and when.

Missed Opportunities asks a different question: which bottles were at peak during a period, were never opened, and are past window now.

And when nothing was lost, it says why rather than showing an empty panel.

Three views in the App, all computed live from the event log:

Cellar health, bottle counts by state at any date
Drink Soon, windows closing within twelve months, each row naming which claim decided it
Missed Opportunities, bottles that were at peak during a period, were never opened, and are past window today

And in Studio, the review queue, where a proposed claim waits for a person.

Code

github.com/kenwalger/Cellar

My Build Process

Tools: Claude Code in a terminal beside WebStorm, with Sanity's MCP server connected so the model could check current documentation rather than recall it. One thing worth stating plainly: the prompts themselves were drafted in a separate planning conversation with Claude in the chat app, where I also made the decisions at each fork before relaying them. Calling this one developer and one agent would be inaccurate.

The project entered implementation with fourteen planning documents: a content model, a temporal resolution specification, a build plan, a seed data plan, and eleven architecture decision records. None of it was code. That turned out to matter, though not in the way I expected.

The pattern that emerged

specification
     ↓
agent conflict pass, no code written
     ↓
human decision
     ↓
implementation
     ↓
independent oracle, measurement, or experiment
     ↓
gate only I can close

Every stage started with a prompt that said, roughly: read the specs, check them against the current platform, report where they disagree, and stop. Every stage found something. The count went up as the project went on, not down. Stage 1 found three spec errors. Stage 4 opened with ten conflicts, two of them spec errors.

Prompts that worked

"Report conflicts and stop before writing code." The single highest-value instruction in the project. In Stage 1 it found that I had specified underscore-prefixed projection fields, which Sanity reserves for system fields, and a validation rule requiring an assessment to have been created in a particular state, which Sanity cannot express because validation sees current state only. Implementing that rule literally would have made the accept transition permanently invalid, breaking the exact workflow the rule existed to protect.

Gates that name what does not count as proof. For the App SDK gate I wrote out the path being tested and then listed the ways it could be faked:

Do not satisfy this gate by calling @cellar/core separately from the App, querying the dataset through another tool, or relying only on the existing test suite. The purpose is to prove the complete path: Content Lake → App SDK → CELLAR_QUERY → toCellarSnapshot → bottleState() → rendered view.

You cannot see the rendered view, so do not report the gate as passed. Give me the command, the URL, and exactly what I should see.

That last paragraph matters more than the first. Claude Code has no browser, and a model asked to confirm something it cannot observe is under pressure to confirm it anyway.

"State the expected result before deploying." For the Functions experiment, the probe had to produce one predetermined log line, written down and approved before anything was deployed. A bundler can resolve an import and still ship a shimmed or stale module; a log line proves nothing on its own.

Oracles, with an explicit instruction not to repair them. Expected outputs were generated before implementation and the prompt said to treat them as an oracle rather than as output to be fixed. When a test disagrees, the cheapest move for a model is to adjust the fixture.

Prompts that didn't, and where I was wrong

I proposed a test that would have written to production. Setting up the Functions probe, I suggested scoping the trigger at the production dataset with a filter on a document type that doesn't exist, to avoid creating a scratch dataset. Claude Code declined, correctly: scoping is safe, invoking is not. A document Function can only be invoked by writing a matching document, so my plan would have put two mutations permanently in production's history to avoid a dataset that deletes whole.

I told it to give up on a test I assumed was impossible. After finding a test that passed while measuring nothing, I said that if the clause couldn't be isolated, it should fall back to a weaker assertion. It ran the experiment instead and found a fixture, on the realization that now in this system is a parameter rather than the clock. "As of June 2024, what had I missed that year?" is an ordinary question for a time machine, and it makes the case constructible.

I approved a feature without asking what it cost. An extra column showing "3 of 5 at peak, 1 opened in time" came back with a measurement I hadn't requested: it costs about four times what the rest of the view does, because it scans every other bottle of every listed wine. Still 5% of a frame, so nothing changed, but the earlier estimate had been for something narrower than what shipped and it said so.

Where the model got stuck

It confidently misread an API. It reported that Sanity previews cannot follow references and deferred a feature to a later stage. Asked to verify against the docs rather than assert, it found the section demonstrating exactly that, said plainly it had been wrong, and fixed it in one line.

It nearly filed a bug that didn't exist. It concluded that installed type definitions contradicted the reference docs about where perspective lives. They don't: the option is inherited through an interface chain that the reference flattens and the .d.ts doesn't. It caught this itself and logged the near-miss instead of the finding. That's the more useful record, because a developer who files bugs that turn out to be their own misreading loses credibility fast.

Right conclusion, wrong reasoning. It argued for inclusive date comparison on the grounds that strict comparison would reject ordinary windows in the seed data. It wouldn't, because same-year windows normalize to 1 January and 31 December, which are strictly ordered. The conclusion was right anyway. A model that is right for the wrong reason is right only until the case changes.

It asserted a gitignore rule instead of checking it. A stage list said the public build's output directory was already ignored. It wasn't: a bare dist pattern matches a directory named dist, and this one is dist-public. Six build files were committed, in the same commit whose accompanying document argued that build output does not belong in a repository. The cause is neat: dist-public was named that way specifically so it could not collide with dist/, and a name chosen to be distinct from dist is by construction one that dist does not match.

It broke an explicit rule twice, and disclosed both times. I told it not to run git commands that change the repository. Twice it ran git checkout -- to undo its own uncommitted edits, and twice it reported this unprompted before I found it. The rule now says explicitly that the file tools are the only way to revert anything, including its own work.

How I course-corrected

Mostly by making verification harder to fake rather than by writing better instructions. The gates got stricter as the project went on: name the path, exclude the proxies, state the expected numbers in advance, and put the confirmation in the hands of the one participant with eyes.

The other correction was measurement. Three of the four real bugs in this project were invisible to every check designed for the feature they lived in.

The App SDK

The gate was one view rendering live production data inside Sanity, with now frozen to a date an independent script had already computed. The numbers had to match exactly: hold 45, drinking 166, past window 33, unassessed 4, consumed 294, 542 total. They did, first try, and I confirmed them in a browser because the model couldn't.

useQuery subscribes, so publishing an assessment in Studio moves the counts in the App with no synchronization layer. Studio edits the graph, the App reads the graph, GROQ traverses it, Functions react to it, and a pure module decides what it all means.

Learning the current App SDK was harder than using it. In one session: Sanity's own bundled agent rule supplied a CLI command with a flag the installed CLI rejects; the quickstart named a template the CLI reference doesn't list; and that template installed the SDK a major version behind the docs I was being pointed at. All three are stale vendor guidance rather than product defects, which is a distinction worth making and also a pattern worth reporting.

The deeper issue was a recurring shape. The documented path is well covered for the common case of rendering and editing documents. Cellar computes an aggregate, and the guidance steered toward useDocuments and useDocumentProjection, which for six numbers would mean roughly 1,500 hook instances and round trips instead of one query. Similar story with useState: the rule correctly warns against holding Content Lake values there, and says nothing about ephemeral view state like a date control. Neither is wrong. But once you step outside the example app, it becomes difficult to tell not recommended from not documented from not supported.

Workflows

An assessment carries a review state: proposed, accepted, or rejected. Only accepted claims resolve windows. An agent proposes; a person decides; rejected claims stay in the dataset, because deleting them would destroy exactly the record this project exists to keep.

Two details I'd defend:

The review state is read-only in the form. A workflow you can bypass with a radio button is decoration. The two document actions are the only path.

An accepted proposal records that a model extracted it. Same tier and same author as a hand-written claim, because accepting it makes it yours, but with sourceMethod: extracted alongside. A system arguing that claims should carry their provenance cannot then be unable to say whether a person wrote this one.

The agent half. A document action on a consumption sends the tasting note, the date, and the wine to Sanity's Prompt API and writes back a proposed assessment. It abstains when the note carries no temporal signal, and abstention writes no document at all rather than a proposal with empty bounds, because a proposal nobody can accept is noise in a review queue.

I checked it against twelve real tasting notes from the dataset, asserting direction rather than exact years, since direction is checkable on a non-deterministic reply. Twelve for twelve, and the outputs were better than the pass rate suggests. Every non-abstaining reply quotes the phrase it relied on, which is what makes the claim auditable: you can see which five words produced a window. Confidence tracks how directly the note states the thing, high on "This is the peak" and low on "Fruit starting to dry out at the edges", which is a hedge in my own note. And on "Could have waited another two years" from a 2024 opening it returned 2026, which is arithmetic rather than classification.

The abstentions are the part I did not expect. They name what the note actually contains: "the note only describes the occasion for opening the bottle and gives no signal about whether the wine was too young, drinking well, or past its best."

One honest result. Hunting for a proposal that would visibly move the counts turned into a search, because the seed data carries an assessment on nearly every wine, so the cases where an agent's claim wins outright are the leftovers. The first candidate lost on recency within its own tier: a personal claim of mine from 2018 outranked a proposal dated 2017. The agent's proposals mostly did not change the cellar, because I had usually already assessed the wine myself. That is the authority rule working, and editing the seed data to get better footage was considered for about a line, then rejected. It is the dataset three oracles were built against.

I verified the workflow against a staging copy of production rather than in a unit test. Two claims written in, neither moving a single count while proposed. One rejected, still moving nothing, though accepting it would have moved five bottles and emptied a wine out of Missed Opportunities. One accepted, moving exactly the five bottles predicted a day earlier.

What I cut, and why

One planned piece did not get built: a Function maintaining derived fields on wines and bottles, so the current state of each would be stored rather than computed.

The reason is that a projection is a cache, and the thing it would cache costs 0.07 milliseconds for all 542 bottles. Measured, not assumed. The App reads the event log directly and recomputes on every date change, which is under half a percent of a frame budget.

The more interesting reason is that a projection cannot do the thing this application exists to do. A stored state is true as of one date, which ADR 0012 forced into the open: any clock-dependent projection has to store the date it was computed for, beside the value. Nothing precomputed answers "what was the cellar on 1 June 1999" unless you precompute every date, and that is not a cache, it is a different data structure.

What would change at scale, since "it is fast enough for 542 bottles" is not an architecture:

The arithmetic is not what breaks first, and the numbers say so. The ledger is 211.9 KB today, about 400 bytes per bottle:

	Today	x100	1.5M bottles
Query payload	211.9 KB	20.7 MB	572.6 MB
Full state tally	0.07 ms	~7 ms	~200 ms

The fetch becomes impractical somewhere around fifty thousand bottles, two orders of magnitude before the scan starts dropping frames. A projection would not have helped with the half that breaks first. The fix is scoping the query, by producer or vintage or a search, and a scoped query already existed in the codebase before the Function was proposed.

Where a Function would genuinely earn its place is a different job. A projection is not only a cache, it is also what makes state queryable. Expressing authority-then-recency resolution in GROQ is possible, and my own spec describes it as possible and unreadable, needing a rewrite the first time a tier is added. So "show me every bottle past its window" would mean a second implementation of the load-bearing rule, in a second language, in a place the tests do not reach. That is why two of the Studio polish items were blocked on this Function, and it is a reason that applies at 542 bottles, not only at a million.

I still cut it, because nothing in the demo reads those fields and the writing was not done. But "we did not need the cache" is the smaller half of the answer.

Getting it in front of someone without an account

This took four prompts to settle and produced the finding I'd most want Sanity to see.

A deployed App SDK app cannot be shown to anyone outside your organization. The Dashboard gates on membership, and a changelog entry lists cross-organization visibility as a bug that was fixed. Hosting the app statically doesn't help either: AuthBoundary redirects a logged-out visitor to a login page before a single query is issued. Asked whether any supported configuration satisfies that boundary without a user token, the model enumerated the entire typed auth surface rather than sampling it. Every route resolves to a user token. The SDK recognizes two environments, a Dashboard iframe and a Studio, and a static host is not one of them.

There is exactly one typed field that would make a static page render by asserting it is a Studio. It was recorded as a near-miss rather than recommended, on three documented grounds including that it asserts something false.

So the public page drops the App SDK and reads the public dataset with a plain client. That touches one file, because everything downstream takes a Cellar object and knows nothing about Sanity, and the views are shared rather than forked.

Then the last surprise. A public dataset is exempt from authentication, not from CORS. An unauthenticated read from a github.io origin returns 403 until the origin is allowlisted. Every anonymous read this project had done went through curl, and curl sends no Origin header, so no CORS check had ever applied. Three sessions had verified the dataset was readable using a tool that structurally cannot test the thing that mattered. The failure would have been invisible until the page went live, because localhost was already allowlisted.

Three distinct access-control mechanisms, then, to get one page working: dataset ACL, organization membership, and origin allowlisting. None of them is discoverable from the surface the other two are configured on.

The answer was in the installed code, repeatedly

A pattern worth naming on its own, because it decided four separate questions.

Two doc pages contradict each other about whether the Prompt action requires a schemaId. Session 8 concluded that one of them is wrong and that reading more carefully would not reveal which. The installed type file reveals which in about a minute: PromptRequestBase declares four members, schemaId is not among them, and it is required four times on the other action types in the same file. The troubleshooting page is not merely wrong, it is backwards. Omitting the field is correct; including it is the type error.

That finding deleted a step I was about to take. A schema deploy had sat on the plan for five days as a production write, purely as insurance against a contradiction the type file settles.

The same thing happened with authentication. The HTTP reference says every endpoint requires bearer auth and says nothing about which clients satisfy it. The answer turned out to be in the AI Assist custom field actions guide, which is not a page about authentication and which demonstrates the pattern twice in complete examples with no token anywhere. The authoritative page created the difficulty; the page that resolves it does so incidentally, by example, with no sentence anywhere stating the fact. That removed the second planned production write.

Add the App SDK auth enumeration and the Functions bundling probe and it is four times in two weeks that the shipped code answered a question the documentation could not. The documentation is not bad. It is describing a platform moving faster than it can.

Two more findings I'd hand to Sanity

Functions bundling is documented as a project-wide choice and isn't. The docs describe TypeScript Functions in pnpm workspaces and non-TypeScript npm projects. This repo is npm plus TypeScript, which is neither. Rather than read more, I deployed a throwaway probe with one predetermined expected result. It worked first try, and inspecting the bundle explained why: the CLI inlined and tree-shook the local workspace package while externalizing the registry dependency. It decides per dependency, not per project. A private workspace package never has to be installable, only resolvable at build time.

A Function that does nothing gets an editor token. The probe constructed no client, ran no queries and wrote nothing. Deployment still provisioned a robot token with editor role and no expiry.

What the AI actually contributed

Not mainly speed. Probably faster, but that's the least interesting result.

It found errors in my specifications. It introduced its own bugs and caught some of them. It used mutation testing twice, unprompted, to prove that tests fail when the code they cover is removed. It refused to fabricate data when the ledger had no column for a link it was asked to infer, and that refusal exposed that one of my own verification scripts was passing on a proxy.

The conclusion I'd draw is narrower than "specs make AI-generated code trustworthy":

Specifications make disagreement observable. Independent witnesses make that disagreement useful.

Sometimes the witness is a CSV generated before implementation. Sometimes a second algorithm, a property test over 17,167 inputs, a benchmark, or the contents of a deployed bundle. Sometimes it's me reading six numbers off a screen, because the model cannot see them.

Four examples of why that matters more than green tests:

A date bug in the time control got January 31 to March 1 wrong. All five of my verification dates passed under the buggy version. The application worked, the screenshots were right, and every date I planned to demo was fine. A property test across every slider position found it.

A test asserting that proposed claims change nothing passed while measuring nothing, because its fixture would have lost on recency whether it was proposed or accepted. What caught it was the companion assertion that accepting must change at least one state. Auditing the rest of the suite for that shape found two more.

Then the same idea from the other direction. A CI check grepping the built bundle for leaked credentials fired on every clean build, because its pattern matched a legitimate export name from @sanity/client. A check that always fails gets waved through and certifies exactly as little as a check that can never fail, and it's worse, because a false positive doesn't look broken. It looks diligent.

And spec-first has its own failure mode, which I ran into late. A polish list drafted from the build plan turned out to have three items already done. A plan describes work that was scheduled, not work that is left.

The finding I keep coming back to is a different shape. A normalization rule written on day one, for a reason that had nothing to do with the interface, turned every drinking window into whole years. Weeks later that rule made a horizon slider useless, because the count could only change once a year. A stage after that, it made the obvious period for Missed Opportunities structurally empty, all year, every year. No document connects those three facts, and no amount of reading would have found them. Measuring the data did, twice, in about fifteen minutes each.

Specifications don't only prevent errors. They reach forward and constrain decisions nobody anticipated making.

Beyond the cellar

Wine makes this a pleasantly low-stakes problem. The pattern applies anywhere "what was true then?" matters alongside "what is true now?"

A museum needs to know which attribution it accepted when an object was exhibited, not only which one it holds today. An HR system needs to evaluate an action against the policy in force when it happened, not the policy as amended since.

The model underneath is the same in both: events record what happened, dated claims record what was believed, provenance records where those claims came from, and resolution decides what was knowable at a given moment.

It is not free. You can never answer "what is the drinking window?" without also answering "as of when?", every read costs a resolution pass instead of a field lookup, and nobody can simply set a value. Most systems should store current state, because most systems are only ever asked about now. The decision worth making deliberately is whether historical truth is a requirement or an accident.

Sanity Project Details

Project ID: aos9nze5 · Dataset: production (public)

The dataset is world-readable, so the content model can be inspected directly. Six document types: producer, wine, bottle, acquisition, consumption, assessment, plus a registered varietal object type. 1,645 documents.

About the data. The cellar is loosely based on a real one. The producers, appellations, and club memberships are real, and some of the history is too, including the 1993 Mouton bought in 1996 and opened twice in the nineties. Everything evaluative is invented. Drinking windows, scores, critic notes, and most tasting notes exist for the demo, and critic assessments are attributed to publications that do not exist. Nothing here should be read as a factual claim about any wine, and nothing attributed to a named producer reflects anything they have actually said.

Agent Session

The curated session is here: Ten conflicts before a line of code. It is a link rather than an embed: the transcript contains 383 @ characters, in package names and file paths, and the editor counts each as a user mention against a limit of ten, so the post would not save.

The full record is in the repository either way: fifteen session logs under docs/friction-logs/, each with the prompts and outputs verbatim, plus what was found and what it cost. If you only read one, read session 3, where ten conflicts came back before a line of code was written.

Addendum: the post's own argument, pointed at the post

A reader noticed something after publication that I had not, and it is the best catch of the project, so it belongs here rather than in a footnote.

The two questions at the top of this piece are not the same kind of question. What did my cellar look like on 1 June 1999 asks when things happened. What did I think that bottle's window was in 2018 asks when the ledger learned it. One date control answers both only while a claim starts counting on the day it is dated.

In the seeded data, it always does. The import wrote all 161 assessments as accepted, with assessedAt as their only date, so every date the demo can be driven to gives the same answer either way. The demo is accurate. It is accurate by accident of the import, not by design.

The agent workflow is where the two come apart. A proposal is dated the day after the bottle was opened, often years ago, and acceptance happens now. On acceptance, the resolver treats the claim as having counted since that past date, so an acceptance today can change what the cellar says about 2018. The Accept dialog warns about precisely this. The resolver has no way to represent the alternative.

The reason is that reviewState is stored current state with no history, which is the exact shape this post spends two thousand words arguing against. A claim either counts or it does not, and nothing records when that became true. It survived twelve architecture decision records because the field reads as workflow rather than as domain data.

The fix is one field: write acceptedAt alongside the transition, backfill it to assessedAt for the imported claims, and resolve on assessedAt <= T && acceptedAt <= T. One slider still answers both questions, because T then means the state of knowledge at T rather than what today's knowledge says about T.

I am not shipping it mid-judging, and it is recorded as ADR 0013. A project whose thesis is to write down what you got wrong should not stop doing that the moment it ships.

Your Metric Is Not Your State

Ken W Alger — Tue, 29 Sep 2026 16:13:00 +0000

What a refractometer taught me about observability

In July, I wrote about a farmer walking a vineyard row at dawn, crushing a grape onto a refractometer prism, and typing "13.5 Brix" into a chat window. The agent on the other end didn't care how the number arrived. To the Digital Scribe, I wrote, a number is just a number.

In the vineyard, that's a defensible position. Fresh grape juice is exactly what a refractometer is built to read.

This fall, I've been pointing the same kind of instrument at juice that's fermenting. It turns out the agent should have cared.

Three batches and a prism

Over the last several weeks I've started three small batches: a blackberry wine that's now bulk aging, a pear wine that began fermenting on September 24, and a fireweed honey mead that got its yeast two days later. They're one-gallon batches, which matters for this story, because at that size every sample you pull is wine you don't get to drink and oxygen you've let in.

So I track them with a refractometer. A few drops of liquid on a glass prism, close the cover, hold it up to the light, read a number off the scale. The number is Brix, roughly the percentage of dissolved sugar. It's fast, cheap, and costs a few drops per reading. During the blackberry's primary fermentation I took readings twice a day, every time I punched down the cap of fruit.

And once fermentation starts, the refractometer is wrong.

The instrument isn't lying, exactly

A refractometer measures how much light bends as it passes through a liquid. Dissolved sugar bends light, so in fresh juice the amount of bending is a good proxy for the amount of sugar. The scale on the instrument is calibrated on that assumption.

Fermentation breaks the assumption. Yeast converts sugar into alcohol, and alcohol bends light too. A few days into primary, the refractometer is reporting the combined effect of the sugar that remains and the alcohol that's been produced, and presenting all of it as sugar. The reading comes out higher than the real sugar content. Taken at face value, it says the fermentation is further behind than it actually is.

To get something usable, you run each reading through a correction formula. Most home winemakers use calculators built on formulas originally developed for brewing, and every one of them needs the same extra input: the original reading, taken before the yeast went in.

Here's what that looked like on the blackberry:

Day	Raw refractometer (Brix)	Corrected value
08/29/2026	20.7	20.7
08/30/2026	18.4	16.5
08/30/2026	17.0	14.3
08/31/2026	13.2	8.1
08/31/2026	10.8	4.1
09/01/2026	8.7	0.7
09/01/2026	8.3	0.1

Nothing is malfunctioning. The instrument is doing exactly what it was built to do. What changed is the system it's pointed at, and a number that was a reliable proxy in one state became a misleading one in the next.

Neither instrument measures sugar

The traditional alternative is a hydrometer, a weighted glass float. The deeper it sinks, the less dense the liquid. Sugar makes liquid denser, so falling specific gravity means sugar is being consumed.

Except the hydrometer doesn't measure sugar either. It measures density, and alcohol is less dense than water, so as fermentation proceeds the alcohol pulls the reading down on its own. A finished dry wine routinely reads below 1.000, lower than plain water, which would be impossible if the hydrometer were really a sugar meter.

So the refractometer measures how light bends and the hydrometer measures how heavy the liquid is. Neither measures what I actually care about: how much sugar is left, whether the yeast is still working, whether this wine is done. I infer those. The instruments give me evidence.

Even a steady reading is ambiguous. The traditional sign that fermentation has finished is the same reading several days running. But a fermentation that has stalled also produces the same reading several days running. Mead is notorious for this: honey has very little natural buffering, the pH drifts down as fermentation proceeds, and the yeast can slow or stop well short of dry. A flat line can mean done or stuck, and the instrument can't tell you which. You tell them apart with other evidence: where the line flattened, what the pH is doing, what you expected to see.

That distinction between evidence and state sounds pedantic until you notice it's one we get wrong in software constantly.

Your dashboard has the same problem

CPU utilization isn't health. A service at 15% CPU can be deadlocked, and one at 90% can be doing exactly what you want. An HTTP 200 isn't correctness; it tells you the server returned a response, not that the response was right. A green health check tells you the health check endpoint answered. Model confidence isn't truth; it's a number the model produced about its own output.

Each of these started as a reasonable proxy under a particular set of conditions. Then the system changed, a new failure mode appeared, or someone started optimizing the number itself, and the proxy drifted away from the state it was meant to represent. The dashboard kept rendering it with exactly the same authority.

That's the refractometer problem. The metric was calibrated for one state of the system and is now being read in another, and nothing on the display tells you so. And like the flat fermentation line, a steady metric can mean two opposite things: a quiet error rate might mean a healthy service, or one that stopped receiving traffic an hour ago.

The failure isn't collecting the metric. It's treating the metric as the state instead of as evidence about the state.

A reading needs its history

Back to that correction formula. To interpret today's reading, you need the reading from before fermentation began. Without it, the current number is close to uninterpretable. The same Brix value could describe a fermentation that has barely started or one that's nearly finished, depending entirely on where it began.

The meaning of a measurement depends on its lineage.

This is where I think observability practice most often falls short. We store the value and the timestamp and call it a record. What we usually drop is everything needed to interpret it later: which instrument produced it, under what conditions, what calibration assumptions it carried, what baseline it should be read against, and what state the system was believed to be in at the time.

My fermentation log keeps the raw reading and the corrected value side by side. The corrected value is what I act on. The raw value is what I re-derive from if I later learn the correction was off, find a better formula, or discover I misread the original. A corrected value on its own is a conclusion with its evidence thrown away.

That's a pattern worth stealing. Store the observation separately from the interpretation. Keep the provenance that makes the observation meaningful. Let state be something you derive, with its evidence attached, rather than something you overwrite. It's the same idea behind the forensic receipt work I've been doing: a claim should carry what it was based on.

Watching isn't free

One more thing the one-gallon batches have taught me. I use a refractometer because a hydrometer needs a much larger sample, and in a small carboy that sample is a real fraction of the batch, and pulling it lets oxygen in. The act of measuring changes the system being measured.

It also means deciding how often to look. The blackberry had to be punched down twice a day anyway, so twice-daily readings were free. The pear and the mead don't need that kind of handling, so I've settled on a reading every couple of days: often enough to catch a stall, rare enough to leave the batch alone. That's a measurement policy, and it means my log has deliberate gaps in it. Those gaps are their own story, and I'll come back to them.

Software has the same trade-off: tracing overhead, probes that add latency, sampling that shifts timing. Usually the effect is small. Sometimes it isn't, and how often to look is a design decision with costs on both sides.

Evidence, then state

Here's what I've taken from a few weeks of squinting at a prism.

Separate what you observed from what you concluded. "The refractometer read 9 Brix" and "fermentation is about two-thirds done" are different kinds of statement. They belong in different places, and they should be allowed to disagree.

Keep the raw reading. Corrections, normalizations, and aggregations are interpretations. If you keep only the interpreted value, you can never revisit the interpretation.

Record the conditions that make a reading meaningful. Instrument, baseline, calibration assumptions, the believed state of the system. A metric without its context is a number waiting to be misread.

Treat state as a derived claim. "Fermentation is complete" isn't a reading. It's a conclusion drawn from several readings over time, plus whatever evidence rules out "stuck," and it should be traceable back to all of it.

Notice when your proxy stops being a proxy. Every metric was calibrated for some range of system behavior. When the system leaves that range, the metric keeps reporting with exactly the same confidence.

The agent in July was right about one thing: it shouldn't matter whether a number comes from a clipboard or a five-thousand-dollar probe. Where it was wrong was in believing the number could travel without its history.

The pear wine is still in primary as I write this. The raw refractometer reading says it has a long way to go. The corrected number says otherwise, and the corrected number is only trustworthy because I wrote down what the juice read before the yeast went in.

The number on the instrument isn't the system. It never was.