Probabilistic agents need deterministic acceptance boundaries

dormitivegit — Thu, 13 Aug 2026 05:56:17 +0000

I have been building with coding agents daily for a while now. They are good. The
problem I keep running into is not that they write bad code — it is that they
change things faster than I can produce evidence of what changed relative to what
I had already accepted.

This is a different problem from correctness, and I think it is being
under-discussed.

Tests answer a different question

When an agent hands back a change, the reflex is to run the test suite. If it
passes, ship it.

Tests answer: does the code do what the tests assert?

They do not answer: what moved, relative to the state I reviewed and accepted?

Those come apart in ordinary situations. An agent refactors three files while
fixing one. A generated migration rewrites a fixture you were treating as a
frozen reference. A tool run mutates a config file nobody was watching. Every
test still passes, because no test was ever written about the thing that moved.
You find out later, or you do not find out.

The gap widens as agents get more capable. A weak agent touches one function. A
strong agent touches whatever it decides is in scope — and "whatever it decides"
is precisely the part you cannot pin down in advance, because that
non-determinism is where the value comes from.

The verification layer should be orthogonal to the agent

The tempting fix is to make the agent verify itself: ask it to summarize its own
changes, or run a second agent as reviewer.

I do not think this works as an acceptance boundary, for a structural reason:
both the change and the verification then come from the same probabilistic
process. When they disagree you learn something. When they agree you have learned
almost nothing, because agreement is exactly what a shared failure mode produces.

What I want is a verification layer with the opposite properties:

Deterministic. Same bounded input, same finding, every time.
Model-neutral. Swapping the agent should not change what acceptance means.
Offline. No network call in the verification path — a verification step that can fail for network reasons is not a boundary.
Machine-consumable. A stable exit code and a structured finding, not prose a human has to interpret.

Note that none of this makes the agent less useful. It stays as flexible and
probabilistic as you like. The boundary is drawn at acceptance, not at
generation.

Freeze, change, verify

The concrete mechanism I settled on is boring, which I take as a good sign.

Freeze a bounded set of sources into a hash-addressed manifest. Let the agent do
whatever it does. Verify against the manifest.

$ assurance corpus freeze ./src --manifest baseline.jsonl
result=PASS
exit_code=0
manifest=baseline.jsonl
write_disposition=CREATED
source_record_count=1

# ... agent runs, edits a file ...

$ assurance corpus verify baseline.jsonl
result=HOLD
exit_code=4
counts={"changed": 1, "match": 0, "missing": 0, "self_ingested": 0, "type_changed": 0}
FINDING {"code":"CI03_SOURCE_CHANGED","severity":"ERROR","message":"filesystem
source bytes changed","path":".../src/greeting.py","location":"source_record", ...}

(Real output, with a few header lines — module id, rule-set version, profile —
elided for width, and the absolute path shortened.)

The two things that matter here are the ones that look least interesting.

The exit code is part of the contract. 4 means an integrity finding
specifically, not "something went wrong." A CI job can branch on it without
parsing anything. Once exit codes are contractual they have to be versioned and
tested like any other public interface, which is a constraint worth accepting
early rather than discovering later.

"Bounded" is doing real work. The manifest covers explicitly supplied roots.
Not the whole machine, not an implicit working directory. An unbounded integrity
check is one that eventually gets disabled because it is too noisy, and a
disabled check is worse than no check because you still believe it is running.

A fair objection at this point: for a clean Git repository, git diff covers a
substantial part of the ordinary file-change case. It genuinely does. The cases
where it differs are when the bounded evidence set is not identical to the
repository — several explicit roots at once, deliberately untracked files,
members inside ZIP archives (read and hashed individually), symlink identity
(the link target path itself is recorded and hashed, which is a different fact
from the target file's contents), and the machine-consumable exit semantics
above. If your evidence set is exactly "tracked files in one repo," use
git diff. It is right there and it is excellent.

Authorization must be out-of-band — a lesson I learned by getting it wrong

This is the part I would most like to pass on, because I got it wrong in public
and the failure mode generalizes well beyond my own project.

Suppose your verification layer checks that a mutation was authorized: the record
claims a decision authorized it, and the tool confirms the decision exists,
covers the same object, and was made by the right authority.

The question is: where does "the right authority" come from?

My first public implementation had the authority identity hard-coded as a literal
string in the validation logic. It worked perfectly — for exactly one person.
Every other user constructing a fully well-formed record got a HOLD, because
their authority identity was not the one baked into the source. Three of the
tool's modules were structurally unusable by anyone but me.

I did not notice, and my test suite could not have told me, because all my
fixtures used the same identity as the code. It surfaced during an independent
adversarial review of the repository, and it surfaced only because the reviewer
built two byte-identical inputs differing in exactly one field and observed that
one passed and one did not. Reading the source had not found it; a green suite
had not found it. A single-tenant constant hiding in validation logic is
invisible from inside your own tests, because your fixtures share the constant.

The obvious repair is to let the input document declare its own authority. This
is worse. If the record under verification names the authority that will be
accepted, then a record can authorize itself:

{
  "authority_identity": "WHOEVER_I_SAY",
  "decisions": [{ "decider": "WHOEVER_I_SAY", "state": "AUTHORIZED" }]
}

The validator dutifully confirms the two agree, and the check has become
decorative. This is the same shape as a certificate that vouches for its own
issuer.

The repair that actually holds is to take the expected authority out of band —
supplied by the caller, at the call site, never readable from the artifact being
checked:

$ assurance check pack.json --authority-id PROJECT_AUTHORITY   → PASS, exit 0
$ assurance check pack.json --authority-id SOMEONE_ELSE        → HOLD, exit 3
$ assurance check pack.json                                    → HOLD, exit 3
                                                                  (fail-closed)

and, importantly:

# pack declares its own authority_identity, no --authority-id given
$ assurance check self-declaring-pack.json                     → HOLD, exit 3

The last two lines are the ones worth arguing about. Missing expected authority
is a HOLD, not a pass-through — an authorization check with no expected authority
has nothing to check against, and defaulting to permissive is how these things
quietly stop working. And a self-declared authority never overrides the
out-of-band value, even when it happens to agree with it.

The transferable lesson is narrow and worth stating plainly: trust anchors do
not belong inside the artifact being verified. The second-order version is the
one that nearly caught me — the naive fix for a coupling problem introduced a
self-authorization hole, and it looked like a clean generalization while doing
it.

The layer should refuse to make the decision

The last design constraint is the one people push back on most, so I will state
it plainly: this kind of tool should not decide whether to accept a change.

Concretely, in mine, risk classification returns a tier and an explicit field
saying the classification is not an authorization. Handoff validation reports
structural observations and explicitly reports that receiver readiness was not
machine-determined. Those fields are not decoration; they exist so that no
downstream automation can quietly read a PASS as a go-ahead.

The reason is not modesty about what software can do. It is that the moment a
deterministic checker is treated as an approval authority, people start shaping
inputs to satisfy it, and you have rebuilt the thing you were trying to avoid —
a probabilistic process optimizing against a proxy. Keeping the tool
descriptive, and keeping acceptance with a person, is what preserves the
boundary's meaning.

Where this leaves things

I do not think "assurance for AI-assisted engineering" is a solved problem, or
that a manifest checker is the whole answer. What I am fairly confident about is
the shape:

probabilistic generation  →  deterministic verification  →  human acceptance

with each stage refusing to do the next one's job. Agents stay flexible.
Verification stays reproducible and inspectable. Acceptance stays with someone
accountable.

It is explicitly not a replacement for Git, for tests, for CI, or for human
review. It sits beside all four.

I built FABLE5 as one
implementation of this shape — a local CLI, Python 3.11+ standard library only,
no network calls, no daemon, no model invocation, Apache-2.0. It is early: a
0.3.0 prerelease with 276 tests and CI across Python 3.11–3.14, maintained by
one person. There is a self-contained runnable example that walks the whole
freeze → change → detect → re-freeze cycle in a disposable temp directory in
about two seconds.

I would rather have the architecture argued with than the tool adopted. If you
think the acceptance boundary belongs somewhere else, or that this is a problem
existing CI already handles, I would genuinely like to hear it.