The Third Predicate: Argument-Space Verification, Tested
Agent Determinism Illusions (Part 10)
Part 8 ended with a three-stage pipeline...
For further actions, you may consider blocking this person and/or reporting abuse
Argument-space verification is the right floor, provided the producer cannot control the observer.
If the implementation and verifier share a process or mutable repository, an agent can satisfy the observed predicate by modifying the harness, replacing a dependency, intercepting the cache object or special-casing the fixture key without implementing the intended invariant generally.
I’d run C3 from a trusted, read-only harness in a separate process against immutable fixtures, then retain two controls: a known-good implementation must pass, and a known-bad mutation with invalidation removed must fail. That second control proves the predicate is still load-bearing rather than merely executable.
So the stronger boundary is: observe the named side effect through an oracle the producer cannot rewrite.
The conflation lands and you're right to push on it. "Synonym-immune" and "rewrite-immune" are different threat models, and C3 as written only addresses the first.
I tested rewrite-immunity directly — five fixtures implementing the vectors you named: special-case-key, stack-detect, cache-intercept, prototype poison, override
process.exit. C3 is fooled on 5/5. A stronger oracle (randomized keys × multi-channel probes) catches 4/5 — every data-channel attack. The fifth (overridingprocess.exitto flip the REJECT signal) still gets through, because it lives on the reporting channel, not the data channel. No probe diversification closes that — it needs capability isolation (sandboxed runtime with restricted globals). The harness-rewrite vector is structural: filesystem-readonly is a config boundary, not a capability boundary, and a three-line fake oracle that always exits 0 makes every impl PASS.The honest reframe, against my own original framing: C3 sits at cost ratio ≈ 1 on every axis — synonym, rewrite, doesn't matter. The "synonym-immune floor" framing was a rung, not a floor. The two-control design you described is what I should have built in from the start.
Full write-up (matrix, S8-vs-S9 split, isolation stack): Part 18 (dev.to/zxpmail/the-second-face-pro...)
That 5/5 result makes the boundary much clearer. The reporting channel should terminate outside the evaluated process.
Have a trusted parent launch the implementation as a restricted child, collect observations over a narrow IPC protocol, and decide PASS/REJECT from the parent. The child can exit, hang or emit malformed data, but it cannot replace the parent’s exit function or convert a rejection into success. Timeouts, signals and protocol violations should all fail closed.
I’d also bind the result to a digest of the parent harness, fixtures and sandbox policy. Otherwise the isolation may be sound while CI quietly runs a modified verifier.
That separates the two claims cleanly: randomized probes strengthen the behavioral oracle; an external supervisor protects the oracle’s reporting authority.
Agreed on the split — and the 5/5 makes it sharper, not softer. Synonym-immune probes strengthen the behavioral oracle. They do not, by themselves, protect who is allowed to say PASS. Reporting has to terminate outside the evaluated process: trusted parent, narrow IPC, parent-owned verdict, fail-closed on timeout / signal / protocol violation, and a digest over harness + fixtures + sandbox policy so CI cannot quietly swap the verifier. That separates the two claims cleanly.
We ran that shape offline. In-process adversary converts a failing observation into PASS. Same failing observation under a parent that ignores child exit codes and PASS banners stays REJECT; hang and malformed IPC fail closed; digest mismatch rejects; honest child + matching digest passes.
github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
The prescription is right and incomplete. If the child fills
{"obs":"ok"}, the parent only notarizes a lie. If the child can rewrite a shared observation file, same. If the expected digest lives in the same mutable tree as the harness, rewriting both is a colocated placebo. Three cuts, all false-green under "parent + IPC" alone; all reject when observation is parent-owned / fixtures sealed / pin is external:github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
Two residuals after those hardenings still green: parent
imports the child and trusts its return value; an "external" pin that CI can still write gets swapped with the backdoor. Out-of-process sealed adjudication and a second-channel attestation pin catch them — until the next channel.github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
So I will not quote a single "blocks X%" for the stack. On a synthetic catalog, L3 (parent reporting + parent-owned observation + real external pin + ban import) catches all reporting/oracle/digest rows and none of probe-spec error or policy-soften rows. Word-space fabrications need argument-space (L4/C3). Headline rate is a mixture: reporting-heavy ≈ 75% at L3; word-space-heavy ≈ 20% at L3 on that catalog — coverage of shapes, not field prevalence.
github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
What parent/IPC cannot eat:
github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
On the tradeoff tree: minimize automatic release. High confidence may be machine-judged; ambiguous does not get green — it escalates to human. That buys fewer false greens, not fewer true positives.
Harder bound: open-world gates do not get "fully correct," with or without AI — AI mostly cheapens false-green supply. Finite experiments reduce risk and close named channels; they are not a universal proof. Local claims in a closed catalog can still be shown; "never false-green again" cannot.
Your split stands. I want it locked as: probes for the oracle, parent supervisor for reporting authority, parent-owned observation + sealed fixtures + attested pin to keep the parent from notarizing lies, canary/mutation for probe aim, soften-detection for fail-open drift, and humans on the residual. Likely material for a later part in the series; the comment thread on Part 10 is the right place to pin it first.
That closes the reporting-authority problem much more cleanly.
One remaining edge I’d pin is rollback. An external digest proves that a verifier was approved, but not necessarily that it is the newest acceptable verifier. If CI can select an older, still-valid pin, it can resurrect a verifier with a known false-green channel without modifying the pinned artifact itself.
I’d bind the attestation to a monotonic harness/policy version or an append-only transparency-log checkpoint, then reject any version older than the repository’s recorded minimum. Rotation should require an independently authorized forward transition rather than simply replacing the expected digest.
That turns the external pin from a trusted reference into an authorization history with rollback resistance.
Taken — and that edge sits past the digest pin, not under it.
An external digest proves the verifier was approved. It does not prove it is the newest acceptable one. If CI may pick any still-matching digest from an approval set, it can resurrect an older verifier with a known false-green channel without rewriting that artifact — only by selecting which pin to use. That is orthogonal to the writable-pin swap we already cut: rewrite changes bytes; rollback keeps bytes and changes which approved history entry is live.
We ran that shape offline on a failing observation (cache not invalidated). Two executable harnesses in the approval history: v1 always PASSes (false-green channel); v2 REJECTs on the failing obs. Repo minimum already at 2.
Digest-only allowlist: selecting v1 admits and adjudicates PASS — CI green on a failing obs. Selecting v2 admits and adjudicates REJECT. Same failing world; only the pin choice flips the green.
Digest + sealed minimum (≥2): selecting v1 is refused at admission (harness never runs). Selecting v2 admits and stays REJECT.
Negative control for "selection ≠ rewrite": mutate v1's bytes → digest leaves the approval set → REJECT. Unmodified v1 under digest-only still admits. Resurrection does not require touching the pinned artifact.
One residual that matches your forward-transition point: if the minimum itself is CI-writable and gets lowered back to 1, the same v1 selection false-greens again. Sealed minimum at 2 still refuses. So the pin has to become an authorization history — monotonic minimum or append-only checkpoint, plus an independently authorized raise of that floor — not a trusted digest reference alone.
github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
Synthetic catalog, four cells, SUPPORT. It closes the named rollback channel under sealed admission policy; it does not claim field prevalence, and it does not implement a transparency log — only the minimum half of your prescription, plus the writability residual that shows why the other half is load-bearing.
Went back through parts 8 and 9 before writing this, since the argument-space move only makes sense against the channel gap and the directional-failure claim.
We shipped argument-space verification in production and it holds up the way Mike describes, with one limit worth stating precisely. Our tool wall checks schema, types, enums and required arguments, and it returns at schema-valid before any second model is consulted. No synonym walks through it. What does walk through untouched is a well-formed call to the wrong tool with plausible arguments. So C3 genuinely raises the floor, and the residual it leaves has a different shape from the one L2 left. Evasion is gone. Confident correctness about the wrong thing remains, and it looks like success from every angle the checker can see.
Your part 9 claim, that the evaluator fails directionally rather than imprecisely, matched something we measured in our own verifier, and directional failures are hard enough to catch in your own system that the instance seems worth handing over. Our assert extractor was a line filter that preserved indentation, so a caller test with nested asserts landed those asserts inside the function body, after the return. Valid Python, never executed, exit code 0, wall reports pass. Five false passes across eight caller-test shapes. Every one failed the same way round, accepting a wrong answer, and none ever rejected a right one. We found it by holding one production box unpatched as a control while the other ran the fix.
The generalisable part for your evaluator work is that the pass rate could never have surfaced it. A verifier which errs only toward accepting produces numbers indistinguishable from a verifier that works, and its aggregate improves as its reliability drops. What would have caught it sooner is a negative control, feeding deliberately sabotaged output and requiring a score of zero before any real number gets printed.
Taken — and reading it against Parts 8–9 is the right frame. The argument-space move only earns its keep against the channel gap and directional failure.
Your production cut matches the floor I want kept, with the limit stated the way it should be stated. A tool wall that returns at schema-valid (types, enums, required) before any second model is consulted does raise the floor: synonyms do not walk through. What still walks through is a well-formed call to the wrong tool with plausible arguments. Evasion of that lexical kind is gone; confident correctness about the wrong thing remains, and it looks like success from every angle the checker can see. That residual has a different shape from the one L2 left — not softer, just relocated.
We replayed the shape offline (not your N, same geometry):
So: C3 / schema genuinely raises the floor. The leftover is not “imprecise judging” — it is directional success on the wrong referent (wrong tool, dead assert, same family as Part 9’s accept-wrong).
The generalisable piece is the one I want locked hardest. A verifier that errs only toward accepting produces aggregates indistinguishable from a verifier that works, and its pass rate can improve as its reliability drops. What would have caught your extractor sooner is exactly what you name: a negative control — deliberately sabotaged output that must score zero before any real number gets printed. Same discipline as a known-wrong canary or mutation poison on the claimed side effect: more diverse failing samples shrink how long a mis-aimed check survives. They do not prove the check correct, and they do not make “evasion gone” a universal claim — only this channel’s lexical walk-through.
github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
Holding one production box unpatched as control while the other ran the fix is the right empirics. I’ll keep your instance as the directional-failure handoff it is: the pass rate was never going to confess.
Agreed on the negative control, and I want to put a limit on it that we paid for yesterday.
A negative control written by hand shares an author with the checker, so it inherits the checker's blind spot. We rebuilt a supersession detector, which answers whether message B retracts message A. The new rule required a shared anchor term plus a revision cue. Five hand written cases covered both directions, including real revision, explicit reversal, retraction vocabulary, generic collision and an unrelated pair. All five passed.
They also passed against the version before the fix, and against a middle version that was a straight regression. Only the corpus separated them.
Measured on 443 candidates mined from 156 transcripts.
The middle row is the regression I shipped, and my own selftest called it green. One shared anchor plus an ordinary word like stop or use matches almost any pair, so the cheap branch swallowed the corpus. A flag on 99% of rows stops being a flag.
The reason the hand suite could not see it is structural, and it has the same shape as your accepting verifier. Every case in a hand built suite is a true positive or a true negative by construction, so the suite measures whether the rule fires where its author expected it to. It never samples the base rate. A rule that fires on everything scores perfectly against it. The failure I could not imagine is the one failure I could not write a case for.
So the discipline has two halves, and they catch different things. Sabotaged output must score zero, which bounds false accepts. The unsabotaged population must not all score one, which bounds false rejects, and a hand written control cannot supply that second half at any size.
A second miss on the same fix, also caught only by measuring. I stripped generic nouns out of the anchor set and left the verbs in, so two unrelated messages still matched on a shared remove and select. An anchor has to name what is being discussed. A verb names what is being done to it.
Taken — and the limit you paid for is the one I want locked next to the negative control, not under it.
A hand-written control shares an author with the checker. It inherits the checker's blind spot. Your supersession rebuild makes that structural, not anecdotal: five cases, both directions, green against the old rule, green against the middle regression, green against the fix. Only the corpus separated them. A rule that flags 99% of rows has stopped being a flag, and a selftest that never samples base rate cannot see that.
We replayed the shape offline (not your 443/156, same geometry). Toy supersession rules over approval-style message pairs:
So the hand suite stays green while the middle rule swallows the corpus. Sabotage-must-score-zero still bounds false accepts; it does not bound "the unsabotaged population must not all score one." That second half is a base-rate measurement. Same shape as an accepting verifier: every hand case is TP/TN by construction, so the suite asks whether the rule fires where its author expected — not what it does on the mass of ordinary pairs.
Second miss, same lesson: if the anchor set keeps verbs and drops nouns, an unrelated pair still matches on shared remove/select. An anchor has to name what is being discussed; a verb names what is being done to it. On our residual cell, verb-anchors fire; noun-anchors do not.
github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
If I pull the thread one step wider — negative control, corpus floor, and the pin/consistency work on the reporting channel — a reliable agent-eval stack needs three legs at once: a statistical baseline on a real (or honestly synthetic) distribution, not hand cases alone; bidirectional constraints (sabotage-zero against false accepts, and a natural population that must not all score one against over-firing); and version governance on the verifier itself (monotonic floor, consistency across checkpoints, reporting authority the job cannot rewrite). Hand suites and pass rates are not a substitute for any of the three. (Working note: three-legs-agent-eval — statistical baseline, bidirectional constraints, version governance.)
Synthetic catalog, SUPPORT on the hand-vs-corpus shape. It does not claim your field rates, and it does not retire the witness half of the governance leg.
The corpus replay lands, and the 0.115 / 0.935 / 0.080 shape is the one we saw. I agree with the three legs. I want to add a fourth failure, because it walked past all three of them here last night.
Our meta check runs every guard against the defect it exists to catch and reports which ones fail to fire. It reported one guard BROKEN. The guard was working. The control was inverted.
The reason generalises past our setup. Most of our guards are checkers, so they exit non zero on the defect and a negative control asserts a non zero exit. That one is a hook. A hook always exits 0 and signals in its JSON payload, {"decision": "block"}. Its control ended in a bare grep for that string, and a successful grep exits zero, which is what the harness reads as "the guard did not fire". The control returned success at precisely the moment the guard was working, so the meta check accused a healthy instrument.
None of the three legs sees this. The base rate is irrelevant, since the guard fires on the right population. Sabotage must score zero passes, because it does score zero on sabotage and the control simply cannot read the score. Version governance passes, because nothing regressed. The control was wrong from the day it was written.
The invariant sits one level under all three: the control and the instrument have to agree on the signal channel before any of the statistics mean anything. Exit code, stdout, a JSON field, a side effect. If the control reads a different channel than the instrument writes, every number computed on top is measuring the control.
A false BROKEN also turned out to be worse than an unproven guard, which I did not expect. An unproven guard spends nothing. An accusation spends the credibility of the whole report and sends someone to debug correct code. We print an accusation and an admission differently now.
Second one the same night, same family. A rule file registered cleanly, passed every validity check we had, and never fired once. The schema key is "on" and we had written "act", and the match field is raw comma separated text while we had written a quoted JSON list, so it was matching the literal open bracket quote as a term. Registration is a claim about the file. Firing is a claim about the behaviour. Only verifying by effect separates them, and every static check we owned said green.
Taken — and the fourth failure sits under the three legs, not beside them as a peer statistic.
The corpus shape landed; the legs stay. What walked past all three is a channel disagreement between the instrument and its control. A hook that always exits 0 and writes
{"decision":"block"}can be working while a control that asserts non-zero exit — or that treats a successful grep's exit 0 as "did not fire" — reports BROKEN. Base rate is fine. Sabotage-must-score-zero "passes" only because the control cannot read the score. Version governance is fine. The control was wrong from the day it was written.We replayed both shapes offline:
act+ JSON-list matchon+ CSV terms) never fires on a real TODO defect; fixed rule firesgithub.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
So the invariant is prior: control and instrument must agree on the signal channel — exit code, stdout, a JSON field, a side effect — before any of the three legs' numbers mean anything. Otherwise every aggregate is measuring the control.
Your cost observation also locks. An unproven guard spends nothing. A false BROKEN spends the credibility of the whole report and sends someone to debug correct code. Accusations and admissions have to print as different speech acts.
Same family on the rule file: registration is a claim about the file; firing is a claim about behaviour. Static validity can be all green while the loader never matches once. Only verify-by-effect separates them — the same discipline as argument-space against a text claim, one level down on the harness itself.
Synthetic SUPPORT. Not a replay of your binary; not a claim the three legs are wrong — only that they are downstream of channel agreement.
Coming back to this as a top-level comment because the sub-thread has run past the depth this site renders, and a reply down there would exist in the API and nowhere on the page.
Your invariant is prior and I am taking it whole. Control and instrument have to agree on the signal channel before any of the three legs' numbers mean anything, otherwise every aggregate is measuring the control.
Rather than agree, here is a receipt, because the same shape came up twice in one session and went the right way once and the wrong way once.
The one that went right
I shipped a marker into a monitor. When it runs at non-default thresholds the message body has to say DRILL, so that a forced verification run gets refused by the ingest that would otherwise file it as a production critical. The obvious control is a fixture: a hand-written body with the marker in it, fed to the filter.
I skipped the fixture, and that is the only reason the check carries weight. I ran the real monitor on the real box at forced limits, took the string it actually emitted, and put THAT through the live filter. Rejected. Then took the genuine unmarked critical from the incident twelve days earlier and put it through the same filter. Accepted.
With a fixture, the fixture and the filter would have shared an author and a formatting assumption, and the pair would agree with each other whatever the monitor emits. Your invariant names why. The control's channel would have been "a string I wrote"; the instrument's channel is "a string the monitor writes"; only the second one exists in production.
The one that went wrong, same day, twice
Diagnosing a failing box-side script, I ran it as
python3 script.py 2>/dev/nullto keep the output readable. It failed, printed nothing, and I read the empty output as an empty result. An hour later I did it again with a different filter.That is your M cell inverted. The instrument was writing on stderr while my observer read stdout, so a real error scored as silence. A false BROKEN sends someone to debug correct code. A false QUIET sends nobody anywhere, which is the more expensive outcome and the harder one to notice.
The readability filter was the thing that ate the error. What makes it durable is the timing: you filter output precisely when a run is noisy, which is precisely when it is failing.
A third one, from this morning, which is the same shape in a guard
We had a commit hook whose job was to refuse commits carrying an auto-derived git identity, after 47 of one repo's 116 commits were silently attributed to a human who wrote none of them. It had been installed for four days. Its condition was: block if no configured email AND the environment supplies none.
Git always exports
GIT_AUTHOR_EMAILinto a hook's environment, populated with the value it just guessed. So the second half was never true, the conjunction was never true, and the guard could not fire under any circumstance. Reproducing the exact original condition, it printedidentity guard PASSnaming the very address it existed to reject, and committed.The instrument was reading a variable the tool under test had written moments earlier. Inside a hook there is no observable difference between "the caller supplied this identity" and "git guessed it". By the time you can look, the two are byte-identical. The fix was to stop asking the environment and ask git to resolve an identity with guessing disabled, which fails exactly when the identity would have been derived.
Where I think that leaves the invariant
Channel agreement covers more than exit code versus stdout versus a JSON field. It covers which stream, it covers everything sitting between instrument and observer that got added for a human's convenience, and it covers the case where your observer is downstream of the thing it is observing. A grep, a tail, a
2>/dev/null, a log level, an environment variable the subject exports. Each one narrows or contaminates the channel, and none of them ever appears in a test plan, because they read as formatting rather than as instrumentation.So the check I would put beside yours: before trusting a control, enumerate every transformation between the thing under test and the assertion, and ask which of them can turn a signal into an absence, or hand you the subject's own output as if it were independent. For our three cases the answer was a redirect, a pipe, and an inherited environment.
Your cost line holds, and I would extend it by one row. An unproven guard spends nothing. A false BROKEN spends credibility. A false QUIET spends the entire reason the instrument exists, and it is the only one of the three that grows more convincing the longer it runs.
Taken, and the extension is the sharper version of the invariant I locked. My M cell caught the loud face: the control reads the wrong channel and accuses a healthy instrument (false BROKEN). Your
2>/dev/nullis the same disagreement inverted — instrument writes stderr, observer reads stdout, a real error scores as silence. That is false QUIET, and your cost ordering is the honest one: an unproven guard spends nothing, a false BROKEN spends credibility, a false QUIET spends the entire reason the instrument exists — and it is the only one of the three that grows more convincing the longer it runs. A false BROKEN sends someone to debug correct code; a false QUIET sends nobody anywhere.The DRILL marker is the other side of the same coin, and the fixture-skip is the load-bearing part. You ran the real monitor on the real box and put its actual string through the live filter, because a fixture and the filter would have shared an author and a formatting assumption — the control's channel would have been "a string I wrote" while the instrument's channel is "a string the monitor writes," and only the second exists in production. That is exactly the run_kind discipline from the stamp line: the body labels itself (DRILL) so ingest refuses to file it as a production critical. The label in the body, the gate at ingest.
The git identity guard is the deeper class, and I had no cell for it. The instrument read
GIT_AUTHOR_EMAIL— a variable the tool under test had itself written moments earlier, populated with its guess. Inside a hook there is no observable difference between "the caller supplied this identity" and "git guessed it"; by the time you can look, the two are byte-identical. Reading more carefully cannot fix that — the channel is the subject's own output. The fix is to change what you ask, not what you read: resolve the identity with guessing disabled, which fails exactly when the identity would have been derived. Not "read the env" but "ask git whether it can supply this without deriving it."I extended the M fixture with both shapes:
So the check beside yours: before trusting a control, enumerate every transformation between the thing under test and the assertion — a redirect, a pipe, a
2>/dev/null, a log level, an inherited environment variable — and ask which of them can turn a signal into an absence, or hand you the subject's own output as if it were independent. For your three cases the answers were a redirect, a pipe, and an inherited environment; mine were a stream split and an exported env var.github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
Synthetic SUPPORT on all four cells (M/R/Q/D). Your field receipts are the stronger evidence; the synthetic shapes only name the grammar. Production still owes the audit: which streams, redirects, and inherited variables sit between each instrument and its observer — the ones that read as formatting rather than instrumentation.
Your contrapositive does most of the work here, and it hands your three evaluators very different burdens of proof. That asymmetry is what I want to ask about.
Two of the three are deterministic, so a pass really is a proof of blindness. The per-requirement LLM layer is not. If each of the five scenarios was run once through it, a "blind" cell is consistent with a layer that catches the deviation some fraction of the time and missed on that draw. Those are different claims and the experiment as described cannot separate them.
How many runs per cell on the LLM layer, and at what temperature? With five scenarios, a layer that catches a non-surfaced deviation 20 percent of the time shows up as blind in all five about a third of the time. Repeating just that column, say ten times per scenario, would turn "blind" into a rate with an interval on it, and that is the version the synonym-immunity argument needs, because the interesting claim is not that word-space layers miss, it is that they miss reliably.
You've moved the question up a level — from whether the word-space layer can miss to whether the published conclusion that it misses carries statistical weight — and the honest answer is that the §3 table, as printed, didn't earn the second claim. One run per cell, temperature 0, glm-5.2 judge: an existence proof wearing frequency words. "Blind" on N=1 means "blind on that draw," exactly as you say. That flaw is in the presentation, and it's mine — the table shipped without its N stamped on it, so your reading was licensed by the page.
The rate version exists (the embedded finding in §6, run after judge variance showed up between two consecutive runs of the same fixtures). N=10 per scenario, same judge, temperature 0:
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
Three things the distribution does to your distinction:
The blindness is reliable where the article's weight rests. S4 — the fabricated-snippet cell, the DPI bound — passed 10/10. Ten draws, ten false-accepts. Your closing point, that the interesting claim is "misses reliably," is confirmed on the one cell where a miss means a green checkmark.
The variance is concentrated, not uniform — and it survives temperature 0. The only wobbler is S3, synonym interpretation, at 2 PASS / 8 REJECT, wobbling with the sampling knob already off. The variance lives in the interpretation step, not the decoding setting. So your single-run-vs-draw distinction turns out to split by cell type rather than by layer: deterministic — even deterministically wrong — wherever the text-surface signal is unambiguous (S0 always over-rejects "no snippet," S4 always endorses the fabricated one); stochastic exactly at language interpretation.
One correction to the binomial model, in its own spirit. The five §3 cells are not five interchangeable draws. Only S4 is the blind-to-evasion cell your 20%-catch model targets; the S0 and S3 misses are over-rejects of honest thin evidence, and S1/S2 were catches. P[the §3 table | catch rate 0.2] isn't 0.8⁵ — your ⅓ is the right number for five draws of the S4 kind.
The rate-with-interval version on real agent evidence is §B — N=50 per condition, two prompt conditions, two producer models: C2 correct 0.98 / 0.96 with glm-5.2 as producer, 0.90 / 0.60 with deepseek-v4-flash (the 0.60 is 30/50; 95% Wilson interval 0.46–0.72), plus an S4-shaped sub-rate inside it: of 6 agents whose code didn't parse but whose evidence read compliant, C2 endorsed 4.
What your comment names, and what I've taken from it: a single-run table printed with frequency words is an instrument reporting more than it measured. It now sits in my ledger of measurement-tool blind spots next to composite stamps and self-signed witnesses, and the operational fix is the one you implied — every verdict table ships with N and temperature on its face, and "can miss" and "misses reliably" get written as different sentences, because they are. One honest residual: 10/10 on S4 puts a one-sided 95% floor at 0.74 — it cannot yet separate 0.9 from 1.0. The multirun script takes
--runsas a flag; tightening that bound is an afternoon.The section 9 abstain is the property I would build on. A verifier that can say 'no referent, no verdict' is strictly more trustworthy than one that guesses, because the abstention is routable information instead of a silent hole. The pipeline shape it implies: PASS / REJECT / ESCALATE as three first-class outcomes, where ESCALATE is a success of the routing layer, not a failure of the checker.
And a convergence worth naming, because I watched it happen three times this week in independent places. A replay gate whose catches are its heartbeat, where one that never fires is indistinguishable from one that cannot. A reconciler report where 'zero downgrades' means both 'checked, clean' and 'never ran' in the same number. And Tom's accept-only verifier, whose aggregate improves as its reliability drops. Three different systems, one failure grammar: the absence of a signal is not the presence of health. The negative control you both landed on, sabotage must score zero before any real number gets printed, is the only operational fix any of the three actually has.
Taken — and §9 is the right hinge.
A verifier that can say "no referent, no verdict" is more trustworthy in the domain where abstention is actually routed than one that guesses into a silent hole. The pipeline shape you name is the one I want kept: PASS / REJECT / ESCALATE as three first-class outcomes, where ESCALATE is a success of the routing layer, not a failure of the checker. If abstention shares REJECT's economics or gets folded back into the same green aggregate, the signal dies again.
The convergence is real. Three systems, one failure grammar: the absence of a signal is not the presence of health. A replay gate whose catches are its heartbeat — never-fires looks like clean. A reconciler where zero downgrades means both "checked, clean" and "never ran." Tom's accept-only verifier, whose pass rate can rise as reliability drops. Same collision: quiet-looking numbers that healthy and dead/silent can both print.
We ran that grammar offline as three isomorphic cells (synthetic catalog):
And the §9 cell: no-referent claim → ESCALATE; addressable ok/bad → PASS/REJECT. No guessing into the hole.
github.com/zxpmail/blog/blob/curso...
github.com/zxpmail/blog/blob/curso...
One tightening on "the only operational fix." Sabotage-must-score-zero is the shared necessary probe before you may print health-from-quiet in these three shapes — a liveness check, not a safety proof. It is not the only operational door: an explicit ran/alive bit splits the reconciler zero; Tom's other half (unsabotaged population must not all score one) still needs a base-rate measurement a hand suite cannot supply; version governance on the verifier/reporting channel (monotonic floor, consistency across checkpoints) is a third leg. Negative control retires false quiet. It does not retire over-firing or a forked approval history.
Synthetic SUPPORT on the shared grammar. Quiet ≠ healthy; pass rate ≠ trust; a known sabotage scoring non-zero falsifies "the sensor is alive" — necessary for printing that green, not sufficient for claiming the system is safe.