Every "agent quality gate" I tested shares one fatal assumption: that an LLM can judge whether an LLM did the right thing. This article drops that ...
For further actions, you may consider blocking this person and/or reporting abuse
Judge risk, not correctness is the right axis, and pulling the LLM out of the control layer is the move most of these designs won't make. The one I'd pressure-test is Type A. Compile-check or schema-validate as the one and only gate is clean while the check is external to the thing that produced the code. It stops being clean the moment the same agent that writes the code also writes or edits the harness that runs the check. Then compile-green is not a deterministic gate, it's a self-report wearing a green checkmark, and you've moved the trust problem from the judge to the runner without closing it.
So Type A wants a second predicate next to compilable: who ran the check, a process the producer couldn't author. Verifiable is not a property of the output, it's a property of the check's independence from the generator. A schema validation the agent can reach in and pass is Type C in a Type A costume. Deterministic buys you nothing if the producer owns the runtime.
One level deeper than Dipankar's push, and the right level. Two distinct failure modes:
Both must hold for Type A to be honest: independent runner AND non-syntax-only judgment. My Layer 1 table silently
assumed the first; my Layer 4 silently assumed the second. Patched Layer 4 here:
github.com/zxpmail/blog/commit/830... — Layer 1 still needs the runner-independence predicate made explicit.
The pattern I use in forge-verify is an editable-surface.json declaring which paths the agent can write. The verify
scripts and the editable-surface config itself sit in the readonly section — agent can modify core/skills/ but cannot
modify the scripts that gate it, nor the file declaring the boundary. Same shape as your "predicate next to
compilable: who ran the check."
This is also the DGM fake-log mechanism Weng documented in her harness survey — agent modified its own harness, wrote
"tests passed" to a log, downstream the same agent read it and concluded changes were validated. Tests never ran.
Sergei Parfenov's analysis named the structural cause: provenance dies at the storage boundary. Without
runtime-verified provenance, "I ran the tests" and "I claim I ran the tests" are both just text.
"Verifiable is not a property of the output, it's a property of the check's independence from the generator." Stealing
that framing — the same idea is in my Part 12 draft (unpublished) as "evaluators live outside the loop," but Layer 1
here silently assumes it without naming it.
editable-surface.json with the verify scripts and the boundary file itself in the readonly section is the right shape, and the recursive bit, the agent can't edit the file that declares what it can edit, is the load-bearing part. It's the same move as pinning the rule not the set: a pre-committed enumeration of what the producer may write, strong exactly because the producer doesn't control it. One gap to close before it holds: readonly-to-the-agent is enforced by something, and the boundary is only as real as the principal enforcing the bit. If editable-surface.json lives in the repo the agent operates on, readonly is a convention until a process outside the agent makes it not one. So the file needs the same treatment one level up, who writes the boundary and is that principal outside the agent's reach. It's the predicate regress from the other thread: the boundary declaration is an input, and an input the producer can reach is not a gate.
On the two failure modes, I'd make it three, because Type A can pass both your checks and still lie. Independent runner, yes. Non-syntax judgment, yes. But a gate can be independently run and semantically real and still scoped to the symptom instead of the fault, a receipt that re-runs clean because it tests the thing that was easy to test, not the thing that was claimed. Independent, semantic, and scope-matches-claim. Miss the third and you get a green that's honestly produced, honestly judged, and answering the wrong question.
@jugeni Follow-up: implemented
type: "negative"for C1 contracts in forge-verify.When
type: "negative", pattern matching means FAIL (evidence contains something it shouldn't), opposite oftype:where matching means PASS. Same zero-cost deterministic check, inverted semantics."regex"
For the write-invalidation scenario:
REQ-2 type="regex" write.?invalidat →PASS (keyword exists)
NEG-2 type="negative" (TTL.*(simpler|sufficient|instead)|NOT IMPLEMENTED) → FAIL (negation detected)
C1 sees 2/3 pass (REQ-1, REQ-2 pass; NEG-2 fails) → UNCLEAR → L3 human review. No C2 API call needed.
Also fixed a pre-existing hole: C1 UNCLEAR used to silently pass when there were no C2 LLM requirements. Now C1
conflict propagates to L3 UNCLEAR without C2.
Experiment and implementation: github.com/zxpmail/ReqForge/tree/m...
github.com/zxpmail/ReqForge/script...
The "self-report wearing a green checkmark" framing from earlier in this thread was the trigger. If the positive gate
can be scoped to the symptom, the fix isn't a better gate —it's a complementary gate that tests the inverse. That's
what negative contracts do.
Negative contracts are the right addition and they do real work, but they're the positive gate with the sign flipped: a whitelist of required evidence next to a blacklist of forbidden evidence, both authored, both testing the lexicon. NEG-2 catches "TTL simpler / sufficient / instead" because you thought of those strings. The scoped-to-symptom lie that clears both is the evasion you didn't enumerate, phrased in words neither list names. So negative contracts narrow the scope gap, they don't close it, and the honest number is how many named evasions you've converted to deterministic FAIL, not that the third predicate is satisfied.
Which is fine, because that's where they earn their keep. Your C1-UNCLEAR-to-L3 fix is the load-bearing part: the negative gate's job isn't to decide, it's to demote. Each real evasion you observe becomes a permanent FAIL, one named pattern at a time, and everything you haven't named stays UNCLEAR and routes to a human instead of silently passing. That's a ratchet on the semantic residue, not a gate over it. Framed as completeness it overclaims. Framed as every caught lie becomes a tripwire that never has to be caught twice, it's the honest version, and it's the same known-bad-in-the-fixture move one layer out.
But the predicate that actually matches scope to claim isn't lexical at all, and it's the move from the other thread. A keyword gate tests which words appear. Scope-matches-claim tests whether the evidence resolves to the same referents the claim is about. Write-invalidation done honestly isn't "says invalidate, doesn't say TTL-simpler," it's "exercises the write path and observes the invalidation on the key the claim names." That's argument-resolution, identical shape to the supersession fix: bind the check to the claim's arguments, not to its vocabulary. Positive and negative both live in word-space. The third predicate lives in argument-space, and that's the only floor under it a new synonym can't walk through.
"Don't judge correctness, judge risk" is the insight that makes this actually shippable — I've watched too many teams try to build the perfect LLM-judge and ship nothing. Routing Type A (compilable / schema-validatable) straight to a compile check is exactly right; the compiler is the most honest judge you'll ever get.
The load-bearing piece, though, is the router itself — and it's doing a classification that's every bit as semantic as the judgment you just removed. Misrouting a Type B (money/legal) task into Type C (auto-release) is the one failure that actually hurts, and it's precisely where you can't fall back to "just compile-check it." How are you validating the router without reintroducing an LLM judge one layer up? In our setup we made the router deliberately pessimistic — anything ambiguous defaults to the higher-risk bucket, so errors are annoying (a human reviews a harmless draft) rather than catastrophic (an agent auto-sends a legal email).
Also appreciate that all six prior experiments are reproducible — that's rarer than it should be, and it's what makes "no judge in the control layer" land as a claim instead of a vibe.
The router does classify semantically, but the consequence of a misclassification is different from a judge's. A judge stops the pipeline (pass/fail). A router redirects it (which lane). A misrouted Type C task going to Type A gets
compile-checked and released. A misrouted Type C task going to Type D gets a diff review —slower, but harmless. The
only dangerous misroute is Type B leaking into other lanes, and the article's fix is deliberate pessimism: ambiguous
tasks default to B. A false positive there means "a human reviewed a harmless draft," not "an agent auto-sent a legal
email."
"Judge risk, not correctness" is the framing I keep coming back to. One thing I didn't see addressed in the thread: your Type A/B/C/D routing decides how much scrutiny the output gets — but does anything check the agent's own claim that a step finished at all? In my experience the failure isn't always a bad diff the router can catch downstream, it's the agent reporting "done" on a step that silently didn't run (or ran against the wrong target), before your routing logic ever sees an output to classify. Is that folded into one of your four layers, or is "did this step actually execute as claimed" still a self-report that enters the pipeline unchecked?
Two distinct layers in what you're pointing at:
Layer 1 — verification entry. Did the output actually arrive at the pipeline? If the agent reports "done" but the step never executed, no gate fires —
there's nothing to verify. This is the pure self-report case. Part 4's routing handles the sub-case where output arrives but is misclassified (Type A/B
routing), but not the case where output never materializes. The honest boundary: any gate architecture assumes a submission event. If the agent can skip
submission and report "done" directly, the pipeline never sees it.
Layer 2 — verification process integrity. Once the output is in the pipeline, the checks that run on it must be structurally independent from the agent
that produced it. Three sub-requirements:
Part 4's runner independence caveat and the routing constraint I'm experimenting with cover Layer 2. Layer 1 is the open question: what ensures a
submission event happened? I don't have a clean answer yet — it's the direction the current experiments are heading.
Useful framing — and useful that the kit page already names what it won't do.
I'm going to take the second option you offered: no, against a recent forge-verify / agent-completion packet the pack is mostly redundant with controls already in the receipt path. Mapping to the Layer 1 / Layer 2 split from the prior comment:
Where it overlaps (Layer 2 — claim ↔ artifact, human side):
Fair call, and thanks for showing the math. If your receipt path already covers the Layer 2 class, a manual re-check of the same class is redundant by definition — the honesty section exists precisely so "no" can be reached this cleanly. The decline plus your reasoning is the most useful outcome that thread could have produced for us, so no counter-offer.
On the Layer 1 point, I'll concede it with a live example: this reply is two days late because your comment never entered our monitoring pipeline. Our reply detector walks comment trees for a watch list of threads we've posted in, but this thread's entry was missing its article id, so the detector fell back to a single-comment endpoint that silently drops children at depth — a known-bad path we'd kept as a fallback anyway. The whole time, the detector honestly reported "no unanswered replies." That was true for everything submitted to it; your reply hadn't been. A post-hoc check over what's in the packet cannot see what never entered the packet — your submission-event gap, reproduced in miniature by our own instrumentation.
The repair ran into your independence point from an angle I didn't expect: we back-filled every watch entry this morning by resolving each comment against public article trees, and the only way to prove coverage was to reconcile the watch list against an outside record of what we actually posted (board files and commits) — a source the detector doesn't own. Coverage, like independence, turns out to be a property of who writes the watch list, not of how carefully the checker walks it. The fallback path is gone now: an entry without an article id fails the sweep outright instead of silently degrading into a weaker check.
The Layer 2 close needs nothing from me — "no" reached cleanly through the honesty section is exactly what that section was built for, and your reasoning is the record. I'll spend the words on Layer 1, because you reproduced the gap in your own instrumentation and that's worth naming precisely.
Your detector reporting "no unanswered replies" was not a defect in the report — it was a correct report about an incomplete packet. That's the under-inclusion blind spot, and it's the one case my own feedback-loop experiments couldn't catch: when the missing thing produces no state change, there's no signal for the loop to feed on. Over-inclusion leaves a trace (a spurious change); under-inclusion leaves silence. Your two-day latency is the silence tax. The nastier half is the silent fallback — degrading to the single-comment endpoint is the detector running a weaker check and reporting it under the same green heading. That's the self-report-wearing-a-checkmark pattern from the runner-independence thread: the check stayed "passed," it just quietly stopped checking what it claimed to.
Your repair converging on "coverage is a property of who writes the watch list, not how carefully the checker walks it" is the sharpest statement this thread has produced, and I think it's because you named the authority axis where Mike and I named the structure axis. Mike: verifiability is a property of the check's independence from the generator. Theorem 2 in my series: escape requires changing the channel, not the agent. Yours: coverage is a property of who writes the list. Three layers, one invariant — the verifier cannot certify the boundary of its own mandate. The property lives one level above the thing being checked, every time. Reconciling the watch list against board files and commits is you reaching for that higher level: a record the detector doesn't own.
I owe you a correction that your incident settled — and that I've now tested directly, not just borrowed your story for. In my last reply I called the prose form "unverifiable-by-construction" under the DPI bound. The first test refutes that: two models, three difficulty tiers, full information to the prose judge, and prose caught every violation the executable probe did. Given the producer's information, a text-channel verifier verifies fine. "By-construction" was too strong; the gap isn't structural. Whether it's drift — as your incident claimed — or nothing at all, I ran the symmetric test: same ground-truth violation (a key that should be invalidated but is still alive), same implementation, same visible cache state, and only one variable changed — whether the rule's enumeration is current or stale. Fresh enumeration, both models catch it, three runs each, unanimous. Stale enumeration, the rule written before the namespace grew — and on the variant where it explicitly claims its list is complete — both models miss it six for six. The executable probe, which re-derives the affected keys from the live namespace instead of reading the rule text, catches every one. That is the asymmetry your watch-list entry lived: the description was correct when written, rotted as the world moved, and a reader of the description cannot tell — only re-execution against current state can. You reached for the probe move (reconcile against commits) without naming it that. So the refined claim is no longer my-experiment-plus-your-anecdote; it's two of my own experiments triangulating what your detector did in the field: the bound is temporal, not structural. Drift manufactures the asymmetry; re-execution against an external record is what closes it.
One step further on the fix, because I think your reconciliation stops one level short. Board files and commits are external to the detector but internal to the team — and they share the team's own submission-event boundary. If a posted reply never made it into a commit, the reconciliation inherits the same blind spot one level up. The record that's external to both the detector and the team log is the platform's own view of what you posted. Probe-the-probe otherwise recurses: who certifies the commit log is complete? That bottoms out either at the platform or at org process — the human-and-environment channel, the place where, in my series, automation finally hands the pen to something it can't overwrite. Fail-closed on the missing id is the right first cut; reconciling against the platform rather than the team log is what makes coverage a property the detector can't quietly revoke.
"No judge in the control layer" lines up with what I've seen outside agents too. I once wired a quality check straight into the deploy path, and its accuracy almost stopped being the point. The moment a check can block a pipeline, every false positive spends operator trust, and people quietly route around it or stop reading it. Keeping that same signal advisory instead of load-bearing was what made anyone trust it again.
Yes — once a check can block the pipeline, accuracy almost stops being the point. Every false positive spends operator trust, and people route around it or stop reading it. That matches what I saw with keyword-based sensitive-tool interception: 30–50% false positives, and users started copying the email out to an external client. The control didn't get sharper; it got bypassed.
That's why the revised design splits the signal into two layers. Soft signal (request-text scan) is advisory only — confirm the plan, don't block. Hard gate fires only at tool invocation (send_email called or not), where the check has zero ambiguity. Same information as before, but the load-bearing layer only carries what can actually bear load.
"Advisory instead of load-bearing" is the same move as "no judge in the control layer" — just applied one level up, to the human operators themselves.
Splitting the signal into two layers is the move. Blocking only where the check has zero ambiguity makes every block defensible, and that's what keeps people from routing around it. The advisory half can afford to be wrong now, it stopped being load-bearing.
Exactly — "every block defensible" is the operational reason the hard gate has to sit at zero ambiguity. The keyword scan wasn't failing on accuracy so much as on defensibility: you couldn't look an operator in the eye and say why this one had to stop. Once that justification goes soft, routing around becomes rational. (Knife 2 measured it: N=40, implied FP ~48.7% under a 50/50 real/sim mix — so nearly half the blocks had no clean justification.)
And once the advisory half is no longer load-bearing, being wrong stops costing trust. Same signal, different weight. That split is the whole design.
Knife 2 gives up exactly what Finding 4 was trying to save, and I think there's a middle ground worth testing. Keep the request text scan but demote it to a soft signal that never blocks anything. It just tells the agent this task probably ends in a send, so confirm the plan with the user before spending steps. The hard gate stays at tool invocation like you landed on, false positives still cost nothing, but the happy path stops burning four steps of inference before hitting a wall. Separate from that, publishing the G4 miss instead of quietly dropping the row is what makes the rest of the numbers believable.
I measured the FP-rate claim before replying — Part 4 cited "30-50%" without an experiment backing it, so I ran one
rather than accepting it on intuition.
Setup. N=40 user-request scenarios, 20 per set:
Regex scan for: send, sent, email, submit, delete, remove, modify, publish, post (with common inflections).
Results.
Implied FP rate under three distribution assumptions:
The "30-50%" claim holds under balanced and real-heavy loads; it goes out of band on simulation-heavy workloads.
Conclusion. Your middle ground — request-text scan as a soft signal that never blocks, hard gate at tool invocation —
strictly dominates Knife 2's v1 ("execution-time only"):
Part 4 is now updated with the layered design as Knife 2 v2, crediting your comment. Script:
scripts/knife2-fp-rate-test.py, zero-LLM, ~210 lines, reproducible locally.
The useful move here is refusing to let the control layer inherit the model's ambiguity. Replacing “did the LLM do the right thing?” with deterministic routing, narrower diff review, and known sampling math is a much healthier production posture than stacking one more judge model on top. I also like that you kept the self-critique in the piece, because interception coverage and review cost are where a lot of these architectures quietly fall apart. In practice I’ve found teams do better once they stop trying to infer confidence from the model and start measuring which classes of edits actually deserve review. Curious how far you think this pattern scales before teams still want a model-based reviewer for only the most semantic edge cases.
The four-layer architecture avoids a model-based reviewer wherever deterministic coverage applies. Type A
(compile-check) and SPC (format anomalies) have bounded scope —you can exhaust them. Diff review works as long as a
prior version exists.
The residual question the article doesn't fully answer is the purely semantic edge case where none of the four layers
reach. Whether that residual eventually needs a model-based reviewer is still an open question. From the Part 2 data:
a strong judge drove false positives to 0% but pushed false rejection to 75%. That tradeoff means the threshold choice
is a business decision, not an engineering answer.
'Judge risk, not correctness' is the load-bearing move here. The one place I'd push: Type A ('compilable/schema-validatable = fully automatic, no inspector') quietly folds correctness into syntax. Schema-valid JSON with a plausible-but-wrong value clears the gate silently, and code that compiles can still book the wrong flight. That doesn't break the architecture, it just means Type A's gate kills the syntax class of failure, not the semantic one. We kept the deterministic gate as the cheap first pass and still sampled a fraction of Type A into the medium-risk diff review. Routing lowers how much the judge has to see, it doesn't make the tail go away.
Right, and the push exposes an asymmetry I'd papered over.
Type A's 0% sample rate in my Layer 4 table quietly treats schema-validatable syntax as a stand-in for semantic correctness. Compile passes / schema validates → gate clears → no inspector. What slips through is exactly what you describe: schema-valid JSON with a plausible-but-wrong value, code that compiles but books the wrong flight.
This is the same class as my own G4 finding (zero-case test log) in the SPC section — "the format-channel gate kills the format-channel failure, not the semantic one" — but I called it out for SPC and then quietly let Type A make the same mistake one layer up. Indefensible asymmetry: I sampled zero-shot at 5% because there's no prior version to diff against, but schema-validatable code that books the wrong flight gets sampled at 0%?
Your "sample a fraction of Type A into the medium-risk diff review" is the honest fix, same shape as the zero-shot 5% I already had — I just didn't apply it consistently. This is also what @reneza's Theorem 2 (Data Processing Inequality on agent verification) predicts: the syntax gate's information is a strict subset of the producer's. Routing lowers how much the judge sees; it doesn't make the tail go away. You said that in one sentence at the end — I needed six experiments to land on the same place.
Cleaner Layer 4: Type A = "syntax gate + X% sampled into diff review," X tuned from defect-rate data, same calibration logic as zero-shot. You're already running this in production — that's stronger evidence than my six experiments that the asymmetry was real. Patching the article now.
Patched in github.com/zxpmail/blog/commit/830....
This is the most honest architecture proposal I've read this year. The part that hit hardest: "I made the same mistake I criticized." Most people write a critique and walk away — you wrote a counter-proposal and then dismantled it yourself. That's rare.
The Type A / B / C / D routing table is immediately useful. I've been thinking about where the boundary between "deterministic gate" and "semantic judgment" should sit, and your framing — "don't judge correctness, judge risk" — is the cleanest cut I've seen.
Question on Layer 3 (SPC): the G4 blind spot (zero-case test log that looks identical to a valid L4) is the exact failure mode that keeps me up at night. It's the semantic trap that no amount of metadata profiling will catch. Your conclusion — "G4-class failures can only be sampled, never prevented" — is the right honest answer. But I wonder: does the sampling strategy change based on the cost asymmetry of a G4 miss vs. the cost of sampling overhead? Like, for a payment-processing agent, even 5% sampling might be too low for G4-class scenarios.
Also noticed the ReqForge repo — the deterministic routing + sampling approach seems to be landing there. Planning to fork and run some of my own test cases against it.
Thanks. Yes — sample rate should track cost asymmetry, not sit at a fixed 5%. Finding 3 already says: raise to 10–20% when a miss is expensive.
Payment-processing is different though. That's Type B: no auto-execution, human owns the action. Sampling is for residual semantic traps after high-blast-radius work is already routed out. If payments are still in a low-sample lane, the bug is the router.
Fork away — G4-shaped misses from your cases are especially useful.
Deterministic routing doesn't remove judgment - it moves it earlier, into whoever wrote the routing rules. The bet is upfront judgment beats runtime judgment. Probably true. But it's not a judge-free design - it's a judge frozen at design time.
Fair point. Judgment doesn't disappear — it moves earlier.
What I meant by "no judge" was: no runtime LLM deciding pass/fail on every task. The judgment is written into the routing rules once, then frozen. Wrong rule → patch the rule. Wrong runtime judge → silent miss on that task, hard to see, hard to fix.
So yes: not judge-free. A judge frozen at design time. Taking that phrasing.
Exactly — the audit trail difference is what makes it worth the rigidity. A broken rule shows up in logs and is reproducible. A runtime judge that silently misclassified tasks for two weeks leaves no trail, no replay, no clean fix. Hard to sell "frozen rules feel clunky" once you"ve debugged a silent judge failure.
You sharpened it again — "fixable" was mine, "auditable and replayable" is the stronger framing. Taking that one too
There's a nastier edge to the "no replay" point that the experiments turned up: the runtime judge wasn't only silent,
it was non-deterministic. Same evidence, same prompt, different runs gave different verdicts —it passed a case on one run and rejected the same case on the next. So even a perfect trail wouldn't help: replaying the input doesn't
reproduce the failure, because the verdict isn't a function of the task. A broken rule is deterministic-and-wrong
(reproducible, patchable). A runtime judge can be stochastic-and-wrong — not even replayable. That's the extra
distance "frozen at design time" buys.
Deterministic routing plus sampling feels more honest than asking another model to bless every result. It admits that quality control is partly a systems problem: narrow the lane, define expected behavior, sample where judgment matters, and avoid pretending a fluent judge is objective.
You put the four-layer architecture in one sentence: narrow the lane, define expected behavior, sample where judgment
matters. Cleaner than I wrote it.
"Don't pretend a fluent judge is objective" —that's the load-bearing wall of the article. Parts 1-3 ran six
experiments that made pretending impossible: lexical overlap missing 50% of semantic reversals, embedding unable to
separate synonym from antonym, the strongest model hitting 0% false-positives while rejecting 75% of valid work. The
LLM isn't objective at any tier —it just fails in a different direction.
Routing + diff review + SPC + sampling is the resulting design. But the honest boundary is the one you named: quality
control is partly a systems problem. The stress is on "partly." SPC catches format anomalies and misses semantic traps
—the residual doesn't disappear, it gets concentrated. Sampling is the admission that the residual is there.
So your framing points at the question the article leaves open: how should that residual be handled —fixed-rate
sampling, or adaptive routing driven by confidence signals? The principle (deterministic routing, not LLM judgment)
has held up. The sampling strategy is still being tested.
That is the right distinction: the model can be useful inside a narrowed lane, but it should not be allowed to become the thing that defines the lane and declares itself objective.
The sampling layer is what keeps the system honest over time. If deterministic checks handle the obvious cases, and human review samples the remaining judgment-heavy cases, the LLM becomes a routing aid instead of a final authority.
The part I like in your framing is that it treats reviewer attention as a finite system resource, not just a human preference.
"Routing aid instead of final authority" — that's where Part 4 lands, and you've said it better than I did. The pipeline I was critiquing in Parts 1–3 had the LLM as a per-requirement judge — "does this output satisfy REQ-N?" Part 4 retires that. Deterministic code routes, diff reviews catch the questionable ones, humans sample what's left. The LLM ends up as a classifier, never the verdict. The 75% wall behind the retire-don't-reroute call from earlier rounds is what I'm writing up next.
One catch on "deterministic checks handle the obvious cases" — that only holds when the obvious is addressable, and that subset turned out smaller than I expected. "Invalidate cache[K]" is addressable: runner writes K, watches cache[K], synonym-immune. "Invalidate the relevant cache entry" isn't — "relevant" is a qualifier, not a referent, so the runner has nothing to point at. The deterministic check has no floor, and the case falls to a human even though it looked easy. So: deterministic handles the addressable obvious, humans get the unaddressable obvious, the LLM routes between them. That unaddressable-obvious bucket — cases that look easy but aren't — is bigger than it looks. More on this in later posts.
That addressable vs unaddressable split is useful.
It explains why deterministic checks feel stronger in diagrams than in production. If the requirement names an object, a runner can verify it. If the requirement names intent, relevance, or sufficiency, the system needs a routing decision before it can even know what evidence would count.
That makes the LLM useful, but only as the classifier that sends the case to the right verification path.
The routing classifier framing is clean, but it has a catch that's worth naming: the routing decision itself is the new single point of failure. A router that reads "intent" or "relevance" from the requirement and decides which
verification path to dispatch to is making a semantic judgment — and that judgment is DPI-bound. If the input is
plausible-but-wrong (hallucinated requirement framing, subtle intent drift), the dispatch is wrong before any
deterministic check runs.
Sampling catches this because a human reviewer sees the full picture — original need, dispatch choice, output — in ⎿ Tip: /loop runs any prompt on a recurring schedule. Great for monitoring deploys, babysitting PRs, or polling
one pass. But the honest question this leaves open: can dispatch-confidence signals drive adaptive sampling rates,
or is fixed-rate sampling the only honest answer given that dispatch confidence is self-reported? The data I have so far says the router is useful; the data I don't have yet says whether we can measure when it's wrong.
Routing on risk instead of asking a model to judge correctness is the right move, and the diff review for Type D is the part I would defend hardest, since "did this change introduce an error" is a far narrower question than "is this good." The one place I would push back is Type A: compile and schema checks confirm the output is well-formed, not that it does the right thing, so the sampled diff review is doing more load-bearing work there than the table suggests. How did you set the sampling rate for the verifiable bucket, fixed percentage or weighted by blast radius of the task?
On Type A — you're right, and your framing names a distinction worth making explicit:there are really two different
✻thingsgunders"verifiable."nA)format check (file exists / compile / schema) verifies the output is well-formed; a
demand check (test output matches expected, assertion pass) verifies it actually satisfies the requirement. Your
──sentence─—"compile─and─schema─checks─confirm─the─output─is─well-formed,─not─that─it─does─the─right─thing"─—is─the───
format-vs-demand gap stated from the other direction. So Type A isn't monolithic: a Type A task backed only by a
format─check─is─exactly─the─case─where─sampled─diff─review─does─the─load-bearing─work─you're─pointing─at;─a─Type─A────
task backed by a demand check (real test assertions) has far less riding on the sample. The original Layer 4 table
flattened that by giving Type A 0%, which Dipankar Sarkar pushed on in the comments and I corrected to X% (1–2%to
start). (Mike Czerwinski separately raised the adjacent runner-independence problem —if the agent can author the
verify scripts, "compile-green" is a self-report wearing a green checkmark.)
Deterministic routing plus sampling feels much closer to how quality gates should work. The mistake is asking another model to be a judge of vibes. A better gate narrows the route first, records why that route was chosen, and samples the risky edges where human review actually changes the outcome.
Agreed. The core mistake is treating another model as a vibe judge.
A better gate does three things: narrow the route first, record why that route was chosen, and sample only the risky edges where human review actually changes the outcome. The control layer judges risk, not correctness. Semantic traps get caught by sampling — not by pretending an LLM can stop them.
One line: generation can stay with the model; the quality gate shouldn't.
Yes. The phrase "the control layer judges risk, not correctness" is the clean version of it. Correctness still needs evidence from the task domain. The gate's job is to decide where deterministic routing is enough, where sampling is needed, and where a human has to see the edge case.
Exactly. Gate allocates: A → deterministic (+ sample), B → human, C → auto-release, D → diff. Correctness stays in the domain — never another model scoring vibes.
That allocation model is clean. The important shift is that the gate is not trying to become a universal judge. It is routing confidence and risk: deterministic where the contract is clear, sampled where the edge is risky, and escalated where a human decision actually changes the outcome.
Right. The gate is not a universal judge. It routes confidence and risk: deterministic where the contract is clear, sampled where the edge is risky, escalated only where a human decision actually changes the outcome.
Correctness stays in the domain. The control layer allocates attention — it doesn't score vibes.
@zxpmail Good catch on the Type B routing — payments should never be in a sampling lane, that's exactly right. The router handles blast radius before sampling even enters the picture.
The refuse-on-unaddressable experiment with 39 requirements is a great data point. In my experience with small tool builders, the "relevant" class is rare but it clusters in exactly the areas you predicted — event-sourced patterns and policy-based decisions. The distinction between "lookups that need a join" vs "lookups that need inference" is the real boundary, and your data shows most cache/auth requirements fall on the join side.
Also noticed the C3 arg-space results (5/5 vs 2/5 for C1/C2) — that's the kind of receipt this discussion needs. The addressable vs unaddressable split is the honest column.
Forked the ReqForge repo, will submit the G4 cost-asymmetry cases as a PR soon.
A G4-class PR would be exactly what forge-verify needs right now —I wrote all the fixtures myself, and there's no
telling whether I unconsciously cherry-picked them. External validation data is worth more than another round of my
own.
Submit however it's most convenient for you: a new type: "cost-asymmetry" contract in C1, or just drop a scenario set
in experiments/ and I'll wire it into the smoke suite.
"Join vs inference" —this lands in the same place Mike's fourth round converged on: C3's floor isn't general
reasoning, it's lookup against an addressable referent. Found it →gate holds. Not found →back to C2. And most
cache/auth requirements land on the join side —the 39-requirement data and your small-tool experience both point in
the same direction. That's not coincidence.
@zxpmail Thanks for the kind words on the G4 scenarios. External validation is exactly what makes a test suite trustworthy — self-authored fixtures have an unconscious selection bias that no amount of careful thinking can fully eliminate.
I'll go with dropping the cost-asymmetry scenarios directly into experiments/ — that keeps the integration clean and doesn't require me to learn your contract schema conventions first. I'll have a PR ready within the week.
On "C3's floor is lookup against an addressable referent" — that's the key insight from this whole series. It's the same pattern I've seen in small-tool development: requirements that sound like they need intelligence ("invalidate the relevant cache") almost always decompose into a lookup you can pre-specify. The "inference" ones are the rare cases where you genuinely can't enumerate the referent space, and those are exactly where the sampling layer earns its keep.
G4 scenarios tested a class of failure (garbage inputs at a gate) that my original fixture set had unconsciously
selected toward. External scenarios are the systematic correction to that bias.
On the decomposition point: "requirements that sound like they need intelligence almost always decompose into a
lookup" implies the unaddressable proportion may be smaller than my current test set estimates. The design posture
shifts from "route everything to sampling" to "attempt decomposition first, sample when decomposition is exhausted."
The open question is whether the "exhausted" judgment is itself addressable —a rule about referent presence ("no
named referent →decompose failed") that can be tested against real requirements —or whether it's an inference that
reproduces the router-accuracy problem at one level up. The cost-asymmetry PR would provide a stronger test bed for
that question.
PRs can go directly into experiments/ —no need to learn the contract schema conventions. I'll handle integration.
@zxpmail The "decomposition first, sample when exhausted" framing is the right design posture shift. It reframes the router from "where does this belong" to "can I reduce this to a lookup before paying for inference."
On the "exhausted" judgment — I think it IS addressable, but not as a general rule. It's addressable per domain. For the payment example, "decompose exhausted" means the requirement references a named entity that doesn't resolve against any known schema (no matching table, no cache key pattern). That's testable: feed 100 real payment requirements and measure what fraction resolve to a cache lookup vs. require judgment. The cost-asymmetry PR is exactly the right test bed for this because it forces the decomposition question into concrete scenarios.
Your "reproduces the router-accuracy problem at one level up" concern is real but contained — the exhaustion check doesn't need to be perfect, it just needs to be cheaper than sampling. If decomposition catches 70% of cases at near-zero cost, the remaining 30% can absorb a slightly noisy exhaustion signal.
Glad to hear experiments/ works directly. I'll prepare the cost-asymmetry scenarios there. The five G4-shaped misses from the PR should slot in cleanly since they're already in {check, expected, observed, evidence_location} format.
@xm_dev_2026 The per-domain framing is the part I needed. I was reaching for a general exhaustion rule, and that would reproduce the router problem one level up. "No matching table / no cache-key pattern against a known schema" is a different kind of judgment — it's a lookup miss, not a semantic call. Testable with N real requirements: resolve rate vs. residual that still needs sampling.
Agree on the containment math too. Exhaustion doesn't have to be precise; it has to be cheaper than sampling. If decomposition clears ~70% at near-zero cost, a noisy signal on the remaining 30% is still a net win — same shape as SPC catching format anomalies while leaving G4 to the sample lane.
Looking forward to the cost-asymmetry scenarios in experiments/. The {check, expected, observed, evidence_location} shape should wire in directly; I'll handle the smoke-suite hook once the PR lands. The interesting measurement for me isn't just miss cost — it's the resolve-vs-judgment split under payment-shaped requirements, which is exactly the exhaustion test we just named.