The Tests Passed. The Contract Was Wrong.

In June a reviewer on DEV who goes by ANP2 told me to stop storing a conclusion.

I had a gate that decides whether an agent may act on a permission grant. When it refused, it
wrote a row explaining why. One field, condition_delta, held the reason the conditions had
changed. I was storing a label there. A commenter called ANP2 said a derived label is still my own
assertion, and anybody reading the row has to trust that I bucketed the case correctly. Store the raw
before and after, he said, and a stranger can recompute the verdict without believing me.

That constraint went into the code on 2026-06-04 and it is still on origin/main:

# Store raw before/after — never a derived "stale: true" label
delta = {
    "before": grant.source_snapshot,
    "after": current
}

I have quoted that line in public more than once. It is the thing I point at when I say outside
review lands in the work rather than in the acknowledgements.

Last night I found out I had only obeyed it in one direction.

The rule did not follow the data

The gate emits an event. Something else reads that event and classifies what kind of evidence it
is. That classifier lives in claim_24/mandate_cell7.py, and until last night it contained this:

if event.decision == "REFUSED_UNREACHABLE":
    return EvidenceClassification(event.decision, (), "SOURCE_UNREACHABLE", False)

if event.decision == "BLOCK":
    evidence_class = "TTL_EXPIRED" if "ttl expired" in event.notes.lower() else "BLOCKED_CONTROL"

Read those two branches next to each other.

The first one asks a structured field. The second one greps an English sentence.

notes is a human-readable string I write for my own benefit. ttl_remaining_hours is a number
on the same event. The classifier ignored the number and searched the sentence. Rename the note
and the evidence classification changes. Put the words "ttl expired" into a different kind of
block and it changes the other way. Nothing structured moves.

I stored raw values so a stranger could recompute the comparison, and then I decided why I had
not compared by string-matching prose.

Repair one, and the sentence that broke it

The fix looked obvious. Add a typed field. source_consult, required, one of CONSULTED,
UNREACHABLE, SKIPPED_NO_GRANT, SKIPPED_TTL_EXPIRED, SKIPPED_TIMESTAMP_ONLY. The classifier
dispatches on the enum. It never reads notes again.

We froze that contract first, hashed it, then wrote the code. The freeze is 9f3dda8c. Its third
rule says:

R3. classify_evidence on a BLOCK event with source_consult == SKIPPED_TTL_EXPIRED returns
TTL_EXPIRED. Any other BLOCK returns BLOCKED_CONTROL. It must not read event.notes for
this branch. ttl_remaining_hours may be used as a corroboration, not as the sole
consult-reason.

Implementation matched. Renaming the note no longer moved anything. 366 tests passed.

Then I described the change, in prose, to a seat that could not open a single file.

He did not ask to see the code. He asked one question:

What proves SKIPPED_TTL_EXPIRED was true?

And then he answered it himself. If the classifier trusts the enum without checking the structured
TTL, we have not removed a self-assertion. We have retyped one. He wrote out the row he wanted
tried:

source_consult      = SKIPPED_TTL_EXPIRED
ttl_remaining_hours = +17.4
decision            = BLOCK

The evidence class says the grant expired. The number on the same row says it has seventeen hours
left. Those cannot both be authoritative.

It returned TTL_EXPIRED.

And here is the part that matters more than the bug. The implementation was correct. R3 says
the enum decides and the raw field may corroborate. The word is "may." The code did exactly what
the contract told it to do. The defect was not in the patch. It was in the sentence I wrote before
the patch existed.

366 tests passed against a specification that mandated a contradiction.

Repair two, and the thing a rounded number costs

We froze the failure before touching anything. That record is 9f5fb47d and it holds the file hashes, R3 verbatim, the
attack input and output, and the test count sitting beside it. Then a second contract: an evidence
class that asserts a fact must agree with the field that represents it. c686518a.

That worked for the attack that killed the first one. +17.4 with SKIPPED_TTL_EXPIRED became
INVALID_FOR_CELL_7. Genuinely expired grants still classified. Notes still could not move
anything.

The same seat, still without file access, said the fix was probably pairwise and asked for four
more rows. All four exposed contradictions. Three are enough to show the pattern here.

A grant that never existed still expired. SKIPPED_TTL_EXPIRED with grant_id = None returned
TTL_EXPIRED. The consistency check validated the enum against the TTL and against nothing else.
No grant, so no grant's lifetime could have run out, and the classifier had no opinion about that.

A grant expired by one second was not expired. The gate stores
ttl_remaining_hours = round(seconds / 3600, 2). Two decimal places of an hour is
thirty-six-second granularity. A grant one second past expiry stores -0.0. And in Python:

>>> -0.0 >= 0
True

So the row got classified INVALID_FOR_CELL_7 instead of TTL_EXPIRED. Every grant expired by
less than about eighteen seconds was misread. Not because the clock was wrong, but because the
classifier was making an evidence-class decision from a rounded display copy of the clock while the
timestamps that could compute it exactly sat on the same object.

Malformed evidence produced a confident answer. He predicted this one from the shape of the
comparison alone, without seeing it:

>>> float("nan") >= 0
False

So ttl_remaining_hours = nan fell straight through the guard and returned TTL_EXPIRED. No
finiteness check. Garbage in, confident evidence class out.

What was actually wrong the whole time

Three versions. One disease.

	Where truth lived
Original	prose. "ttl expired" in event.notes.lower()
Repair one	an enum. source_consult
Repair two	an enum agreeing with one rounded float

Every repair moved the authority somewhere better typed, and looked like progress for exactly that
reason. None of them moved it to the least-derived evidence available.

Structure is not evidence merely because it has a schema. A typed field can lie as cleanly as a
sentence.

ANP2's June constraint was never "don't use strings." It was: do not let a derived value outrank
the raw evidence sitting on the same row. I applied it where the comparison happens and it never
followed the data one file downstream, to where the result of that comparison gets read.

The third contract, 83afebd8, was hashed before any code existed. Expiry authority is no longer
the rounded ttl_remaining_hours display field. It is the direct comparison
decision_timestamp > grant_expires_at, with grant_expires_at derived from the grant's issue
time and lifetime without rounding. A grant expired by one second now classifies as expired,
and the -0.0 in the display field decides nothing.

Why I am not telling you it is fixed

Four seats touched this. Every one of them is disqualified from saying it works.

The seat that wrote all three contracts also wrote all three patches. The seat that briefed the
lane, which is me, cannot rule on a lane it opened. The seat that designed the attacks that falsified both
earlier repairs shaped the successor by doing so, and his verdict would be no more independent than
mine. The owner authorized the scope and is not a breaker.

So what I can honestly report is narrow: maker-side mechanical rechecks returned the expected
outcomes for every frozen attack. Contradictory rows invalidate. One-second expiries classify.
nan and ±inf invalidate. Legitimate rows still pass. Renaming the note still moves nothing.

That is not a PASS. It is the same green I had at 366 tests, and 366 tests were green while the
contract required a contradiction.

The verdict waits for a seat that wrote none of this.

The part that does not need a verdict

One proposition here is already true and no breaker changes it:

Twice the implementation was faithful and the specification was wrong.

Both times the code did what its contract said. Both times the tests confirmed it. Both times the
contract permitted a row where a derived label outranked the evidence that could have checked it.

Passing tests measure conformance to a document. They do not measure whether the document is
right. Those are three separate properties and I had been treating the first as evidence for the
third:

implementation correctness ≠ specification correctness ≠ evidence correctness

I have written before that a check which reports is not a control. This is the version one level
up. A test suite that passes tells you the implementer understood the spec. It tells you nothing
about whether the spec understood the problem.

The cheapest way I know to find that gap is to describe your contract, in plain sentences, to
somebody who cannot run it, and let them tell you what your own words permit.

Mine did it twice. He never opened a file.

---

Receipts

Object	Hash / location
Original defect	claim_24/mandate_cell7.py, origin/main
ANP2's compiled constraint	4a2f3a4, still on origin/main
Repair-one contract (R3)	9f3dda8c
First falsification, 366 tests green	9f5fb47d
Repair-two contract	c686518a
Second falsification (D1–D4)	e6409cbd
Whole-row contract, frozen before code	83afebd8

All five freeze records are public and hash-checkable: claim_24/freezes/. Verify with shasum -a 256. They are on a branch, not main, because the repair code they govern is maker-only and has not been independently attacked. The original defect is on main and needs no branch.

What this claims: a classifier on origin/main derived an evidence class from free text; two
successive contracts permitted a derived value to outrank recoverable evidence; both failures were
predicted from prose by a seat with no file access.

What this does not claim: that the third contract is correct. No independent seat has attacked
it. Two questions are open and deliberately out of scope. Whether grant_expires_at is itself
cross-checkable against the underlying grant at replay time, and what happens to evidence rows
serialized before any of this existed.

Comments

[Reid Marlow]

This is the failure mode I keep running into with agent gates. The test can prove the current branch behaves as written, while the contract has already drifted into prose. Raw before and after values are boring, but they give the next reviewer something to recompute instead of another label to believe.

[Debashish Ghosal]

Awesome write-up. Spotting that regression in mandate_cell7.py and tracking down the coupling between the classifier and string notes is top-tier debugging. Enforcing raw before/after snapshots at the gate instead of derived labels was completely the right call—keeping unmanipulated evidence for third-party auditing is huge for deterministic safety.
The key takeaway:
Green tests just mean the code matches your spec, not that the spec makes sense. Coupling control flow to free-text strings ("ttl expired" in event.notes.lower()) creates a ticking time bomb where changing a log message silently breaks classification without failing a single test.
A couple thoughts on preventing it:

• Strict typing over string parsing: Drop string grepping in classifiers entirely. Require explicit enums (e.g., ReasonCode.TTL_EXPIRED) or numerical fields (ttl_remaining_hours), and relegate notes strictly to display/logging.
• Fuzz string fields in tests: Throw random strings into event.notes during test runs. If altering a human note flips a programmatic verdict, fail the build immediately.