I have been building with coding agents daily for a while now. They are good. The
problem I keep running into is not that they write bad code — it is that they
change things faster than I can produce evidence of what changed relative to what
I had already accepted.
This is a different problem from correctness, and I think it is being
under-discussed.
Tests answer a different question
When an agent hands back a change, the reflex is to run the test suite. If it
passes, ship it.
Tests answer: does the code do what the tests assert?
They do not answer: what moved, relative to the state I reviewed and accepted?
Those come apart in ordinary situations. An agent refactors three files while
fixing one. A generated migration rewrites a fixture you were treating as a
frozen reference. A tool run mutates a config file nobody was watching. Every
test still passes, because no test was ever written about the thing that moved.
You find out later, or you do not find out.
The gap widens as agents get more capable. A weak agent touches one function. A
strong agent touches whatever it decides is in scope — and "whatever it decides"
is precisely the part you cannot pin down in advance, because that
non-determinism is where the value comes from.
The verification layer should be orthogonal to the agent
The tempting fix is to make the agent verify itself: ask it to summarize its own
changes, or run a second agent as reviewer.
I do not think this works as an acceptance boundary, for a structural reason:
both the change and the verification then come from the same probabilistic
process. When they disagree you learn something. When they agree you have learned
almost nothing, because agreement is exactly what a shared failure mode produces.
What I want is a verification layer with the opposite properties:
- Deterministic. Same bounded input, same finding, every time.
- Model-neutral. Swapping the agent should not change what acceptance means.
- Offline. No network call in the verification path — a verification step that can fail for network reasons is not a boundary.
- Machine-consumable. A stable exit code and a structured finding, not prose a human has to interpret.
Note that none of this makes the agent less useful. It stays as flexible and
probabilistic as you like. The boundary is drawn at acceptance, not at
generation.
Freeze, change, verify
The concrete mechanism I settled on is boring, which I take as a good sign.
Freeze a bounded set of sources into a hash-addressed manifest. Let the agent do
whatever it does. Verify against the manifest.
$ assurance corpus freeze ./src --manifest baseline.jsonl
result=PASS
exit_code=0
manifest=baseline.jsonl
write_disposition=CREATED
source_record_count=1
# ... agent runs, edits a file ...
$ assurance corpus verify baseline.jsonl
result=HOLD
exit_code=4
counts={"changed": 1, "match": 0, "missing": 0, "self_ingested": 0, "type_changed": 0}
FINDING {"code":"CI03_SOURCE_CHANGED","severity":"ERROR","message":"filesystem
source bytes changed","path":".../src/greeting.py","location":"source_record", ...}
(Real output, with a few header lines — module id, rule-set version, profile —
elided for width, and the absolute path shortened.)
The two things that matter here are the ones that look least interesting.
The exit code is part of the contract. 4 means an integrity finding
specifically, not "something went wrong." A CI job can branch on it without
parsing anything. Once exit codes are contractual they have to be versioned and
tested like any other public interface, which is a constraint worth accepting
early rather than discovering later.
"Bounded" is doing real work. The manifest covers explicitly supplied roots.
Not the whole machine, not an implicit working directory. An unbounded integrity
check is one that eventually gets disabled because it is too noisy, and a
disabled check is worse than no check because you still believe it is running.
A fair objection at this point: for a clean Git repository, git diff covers a
substantial part of the ordinary file-change case. It genuinely does. The cases
where it differs are when the bounded evidence set is not identical to the
repository — several explicit roots at once, deliberately untracked files,
members inside ZIP archives (read and hashed individually), symlink identity
(the link target path itself is recorded and hashed, which is a different fact
from the target file's contents), and the machine-consumable exit semantics
above. If your evidence set is exactly "tracked files in one repo," use
git diff. It is right there and it is excellent.
Authorization must be out-of-band — a lesson I learned by getting it wrong
This is the part I would most like to pass on, because I got it wrong in public
and the failure mode generalizes well beyond my own project.
Suppose your verification layer checks that a mutation was authorized: the record
claims a decision authorized it, and the tool confirms the decision exists,
covers the same object, and was made by the right authority.
The question is: where does "the right authority" come from?
My first public implementation had the authority identity hard-coded as a literal
string in the validation logic. It worked perfectly — for exactly one person.
Every other user constructing a fully well-formed record got a HOLD, because
their authority identity was not the one baked into the source. Three of the
tool's modules were structurally unusable by anyone but me.
I did not notice, and my test suite could not have told me, because all my
fixtures used the same identity as the code. It surfaced during an independent
adversarial review of the repository, and it surfaced only because the reviewer
built two byte-identical inputs differing in exactly one field and observed that
one passed and one did not. Reading the source had not found it; a green suite
had not found it. A single-tenant constant hiding in validation logic is
invisible from inside your own tests, because your fixtures share the constant.
The obvious repair is to let the input document declare its own authority. This
is worse. If the record under verification names the authority that will be
accepted, then a record can authorize itself:
{
"authority_identity": "WHOEVER_I_SAY",
"decisions": [{ "decider": "WHOEVER_I_SAY", "state": "AUTHORIZED" }]
}
The validator dutifully confirms the two agree, and the check has become
decorative. This is the same shape as a certificate that vouches for its own
issuer.
The repair that actually holds is to take the expected authority out of band —
supplied by the caller, at the call site, never readable from the artifact being
checked:
$ assurance check pack.json --authority-id PROJECT_AUTHORITY → PASS, exit 0
$ assurance check pack.json --authority-id SOMEONE_ELSE → HOLD, exit 3
$ assurance check pack.json → HOLD, exit 3
(fail-closed)
and, importantly:
# pack declares its own authority_identity, no --authority-id given
$ assurance check self-declaring-pack.json → HOLD, exit 3
The last two lines are the ones worth arguing about. Missing expected authority
is a HOLD, not a pass-through — an authorization check with no expected authority
has nothing to check against, and defaulting to permissive is how these things
quietly stop working. And a self-declared authority never overrides the
out-of-band value, even when it happens to agree with it.
The transferable lesson is narrow and worth stating plainly: trust anchors do
not belong inside the artifact being verified. The second-order version is the
one that nearly caught me — the naive fix for a coupling problem introduced a
self-authorization hole, and it looked like a clean generalization while doing
it.
The layer should refuse to make the decision
The last design constraint is the one people push back on most, so I will state
it plainly: this kind of tool should not decide whether to accept a change.
Concretely, in mine, risk classification returns a tier and an explicit field
saying the classification is not an authorization. Handoff validation reports
structural observations and explicitly reports that receiver readiness was not
machine-determined. Those fields are not decoration; they exist so that no
downstream automation can quietly read a PASS as a go-ahead.
The reason is not modesty about what software can do. It is that the moment a
deterministic checker is treated as an approval authority, people start shaping
inputs to satisfy it, and you have rebuilt the thing you were trying to avoid —
a probabilistic process optimizing against a proxy. Keeping the tool
descriptive, and keeping acceptance with a person, is what preserves the
boundary's meaning.
Where this leaves things
I do not think "assurance for AI-assisted engineering" is a solved problem, or
that a manifest checker is the whole answer. What I am fairly confident about is
the shape:
probabilistic generation → deterministic verification → human acceptance
with each stage refusing to do the next one's job. Agents stay flexible.
Verification stays reproducible and inspectable. Acceptance stays with someone
accountable.
It is explicitly not a replacement for Git, for tests, for CI, or for human
review. It sits beside all four.
I built FABLE5 as one
implementation of this shape — a local CLI, Python 3.11+ standard library only,
no network calls, no daemon, no model invocation, Apache-2.0. It is early: a
0.3.0 prerelease with 276 tests and CI across Python 3.11–3.14, maintained by
one person. There is a self-contained runnable example that walks the whole
freeze → change → detect → re-freeze cycle in a disposable temp directory in
about two seconds.
I would rather have the architecture argued with than the tool adopted. If you
think the acceptance boundary belongs somewhere else, or that this is a problem
existing CI already handles, I would genuinely like to hear it.
Top comments (3)
Had the same issue. I'd deployed an agent that was quietly rewriting Neo4j edges during retrieval passes, and tests passed because the fixtures were pre-baked snapshots rather than live reads. It's exactly the correctness-vs-drift gap you're naming. The bounded scope part matters a lot; in my case it crept from "just the src/ dir" to "everything except node_modules" within two sprints, at which point the manifest was catching basically nothing.
Strong separation of generation, verification, and acceptance. I’d add one more boundary: byte-level integrity and semantic admissibility should produce different findings. In a research pipeline, a source can be byte-identical yet stale for the current as-of date, or legitimately changed because an official filing was amended. I would freeze evidence metadata alongside the corpus—source URI, retrieval timestamp, effective date, parser version, and schema—and return separate integrity, freshness, and lineage statuses. That lets the human reviewer distinguish unauthorized mutation from expected evidence evolution. Have you considered versioning a policy manifest alongside the corpus manifest so permitted roots, freshness windows, and parser versions are reviewed rather than embedded in the verifier?
This is the same structural lesson as the authority bug, one level up.
Freshness windows and parser versions are policy, and policy hard-coded in a
verifier is invisible from inside your own tests for exactly the reason a
hard-coded authority identity is. It also lands on a boundary the repo declares
rather than hides — docs/LIMITATIONS_AND_FUTURE_SEAMS.md opens with "Hashes
prove byte identity, not semantic truth or provenance."
Concretely, the corpus module (PM-04) answers both of your cases correctly and
helps the reviewer in neither, in opposite directions. A source that is
byte-identical but stale verifies clean — correct about the bytes, silent about
admissibility. A legitimately amended filing raises
CI03_SOURCE_CHANGED—also correct about the bytes, and equally silent about whether that change was
expected. The finding vocabulary has one axis, so nothing in the output
distinguishes unauthorized mutation from expected evidence evolution.
Partial correction, since some of what you're asking for is already there:
permitted roots and exclusions are frozen into the manifest header alongside
the rule-set version, so bounded scope is reviewable as data, not embedded in
the verifier. What's entirely absent is anything time-shaped — source records
carry root, relative path, size, and sha256, with no retrieval timestamp, no
effective date, no notion of parser or source schema. (
schema_versionexists,but it versions the manifest format itself.) So the policy manifest you
describe is skeletal for scope and absent for freshness and lineage.
The boundary I'd want kept sharp is that byte integrity is intrinsic to the
frozen evidence, while freshness is contextual — "stale for the current as-of
date" is a question about something outside the evidence set. That doesn't make
it non-deterministic; it makes the context an input. The check stays
reproducible only if the as-of date is an explicit call-site parameter, never
read from the artifact — same rule as
--authority-id. And note the recursionyour proposal implies: the policy manifest then needs its own trust anchor,
also supplied out of band, or the self-authorizing artifact has just been
rebuilt one level up.
Given that input, the three-way split looks right, and part of it already fits
the existing model: freeze is no-clobber by construction — writing different
bytes to an existing manifest path is a collision finding, not an overwrite —
so an amended filing has to become a new manifest artifact rather than mutate
the old one. What's missing is that the tool reports the drift as an integrity
event with no vocabulary for calling it a lineage one.
That's a larger change than adding fields, so I'd rather think it through than
bolt a timestamp onto a source record. But integrity / freshness / lineage as
separate finding families rather than one validity bit — yes.
Curious how you handle the as-of date today, if you have a pipeline doing this:
pinned per run, or derived from the filings themselves? That choice seems to
determine most of the rest.