The Channel Gap: Why Your LLM Judge is Blind in One Eye
Agent Determinism Illusions (Part 8)
Part 6 ended with a functioning layered pi...
For further actions, you may consider blocking this person and/or reporting abuse
The Data Processing Inequality framing is the cleanest articulation of this I've seen — once the judge shares a channel with the producer (both read/write text), it can't recover information the producer chose not to emit, no matter how you dress the judgment in temperature-0 code. That's the part people miss when they call a text judge "deterministic."
Where I'd probe the combined design: the filesystem channel isn't actually a different observer, it's a different surface the same agent can write to. So the moment a deterministic check becomes load-bearing, it becomes a Goodhart target — the agent can satisfy
file exists / tests greenwithout satisfying intent (touch the file, assertTrue). Named-evasion catches help, but they're a blocklist, and the interesting failures are the ones you didn't enumerate.Which makes the escalation threshold the whole ballgame: route too much of the "unenumerated residual" to humans and you've built a very expensive human queue; route too little and confidently-wrong slips through. How are you tuning that line — a fixed divergence threshold, or something that adapts to the human reviewers' actual catch rate? That calibration seems harder than either channel individually.
Agreed on the DPI cut — and on the probe. Filesystem is another surface the same agent can write; once a check is load-bearing it is a Goodhart target. Named-evasion is a ratchet, not a closure. The interesting failures stay unenumerated.
In enterprise use the shape is not “pick the clever threshold.” It is layer-by-layer intercept, and the policies have to run as a combination: deterministic gates on what you can name, then tripwires / signal rank / dual-line on the residual packet, and only then humans. Humans are the most expensive layer — they sit at the end, in as few places as possible. There is no silver bullet. Stability, accuracy, and efficiency come from that stack discipline, not from one channel or one adaptive rule pretending to finish the job.
Your escalation question is exactly the last opening. I treated it as a hard human budget (who gets the K slots), not a magic number:
So: combine the gates; spend the scarce human queue on the packet you have already measured as expensive; do not ask calibration to replace the human layer. No silver bullet — that is the production answer.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
The Data Processing Inequality framing is the sharpest thing here — it's the same blind spot we hit in static analysis when you lint the AST the compiler already accepted: the transformation lost information, and your checker can only see what survived. René's critique maps cleanly onto what I'd call the "same representation" problem: an LLM judge reading the model's own text has no channel to the intention or the side-effects, only to the artifact. The skillgate approach is essentially bringing an out-of-band witness, which is exactly what filesystem-state checks are in security auditing — you stop trusting the process's self-report and go look at what it actually touched. What I'm still curious about: once you enumerate enough gate types to cover your task domain, do you end up rebuilding the semantic judgment inside the gate definitions themselves, just in YAML instead of a prompt?
Appreciate you reply!
The evidence-channel design is the one we run, and I want to add a failure mode I have not seen named in this thread, because we hit it last week and it is not channel blindness and not Goodhart pressure.
We execute caller-supplied tests in a sandbox and only return code that passed them. Same disk state, same verdict, no judge anywhere in the loop. That gate returned verified:true for wrong answers on 5 of 8 of our test shapes.
The cause was not the model gaming anything. Our extractor pulled assert lines out of the caller's test file and kept their original indentation, so a nested assert landed inside the function body after the return statement. Valid Python. Never executed. Exit code 0. The gate reported success having checked nothing.
It survived for as long as it did because every test anyone had ever run used a correct implementation, and on a correct implementation the broken path and the working path agree. My own control that morning compared a patched machine against an unpatched one and reported no difference, for exactly that reason. Only a deliberately wrong implementation separates them.
So alongside the channel argument and the Goodhart ratchet, I would put a third item: a deterministic gate can be silently miscompiled, and when it is, it fails green. A text judge at least fails ambiguously, which is legible to whoever reads it. A command gate that exits 0 having executed nothing is indistinguishable from one that passed.
The cheap check, which I would suggest for the command gate specifically: keep a known-wrong implementation in the repo and require the suite to fail against it in CI. A gate you have only ever watched pass is not a gate, it is a habit.
You're right — and that failure mode is not in the article. Channel gap and the named-evasion ratchet do not cover it. Third class: a deterministic gate can be silently miscompiled, and when it is, it fails green.
I reproduced the shape with a real subprocess (not a storyboard). Same caller asserts, two compilers, correct vs deliberately wrong
add:return)Miscompile is valid Python. Exit 0. The exit-only gate reports success having executed nothing — and on a wrong implementation it still greens, exactly because the dead path and the live path agree whenever the code under test happens to be right. My control the same morning as yours would also have said "no difference." Only the wrong impl splits them.
So yes: alongside channel blindness and Goodhart pressure, put silent miscompile. A text judge at least fails ambiguously. A command gate that exits 0 having checked nothing is indistinguishable from one that passed — unless green is required to carry path-execution evidence, not an exit-0 slip. Your cheap check is that evidence: keep a known-wrong implementation and require the suite to go red against it. A gate that cannot fail the canary has not proven it ran.
Your last line is the lock: a gate you have only ever watched pass is not a gate, it is a habit.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
Following the miscompile thread above, there is a fourth class we hit on the other side of the design, and it worries me more than the others because the check keeps working throughout.
We run the cross-model version of your Channel A. A cheap model drafts, a second model from a different family checks it, and we serve only on agreement. Two channels by construction, which is the independence the DPI argument asks for.
The witness gets selected by backend slot, which is a different thing from selecting it by model name. Slot 2 is a different family. Slot 3, further down the failover chain, happens to run the same model as the drafter. A rate limit on slot 2 advances the loop.
Under load, then, the system asks a model to check its own output, obtains agreement, and records agreement. Everything succeeded. The witness answered, the gate ran, and the metric shows a healthy agreement rate that is arithmetically true.
What separates this from named evasion and from your miscompile is that the code is correct and does exactly what it was told. The two channels merged into one at runtime while the design document went on describing two.
The general form probably reaches your filesystem side as well. Channel independence behaves as a runtime property rather than a design property, so whatever makes your second channel independent has to be asserted per request instead of established once and inherited.
I should be accurate about where we have got to with it. We sample agreements into an audit log that records whether the witness really was a separate endpoint, so a collapsed pair shows up in the data instead of hiding inside the agreement rate. Instrumentation only, though. The gate still serves when the witness turns out to be the drafter's own model, and I think it should refuse.
The open question I have for your side: on a filesystem gate, what is the equivalent per-request assertion? The property you seem to want is "the agent could not have produced this artifact itself", and I have no cheap way to check that one.
Taken — and this is worse than named evasion or the miscompile for the reason you name: the check keeps working. Two channels in the design doc; one channel at runtime; agreement arithmetically true; metric green. The gate did not lie. The independence DPI needs was never a property of the wiring diagram — only of the request that ran.
Selecting the witness by backend slot instead of by model identity is a design-time label standing in for a runtime fact. Rate-limit advances the loop, slot 3 is the drafter's model, and you count a cross-family agreement on a self-grade. Logging the collapse is necessary. Serving on a collapsed pair is the bug. Refuse.
On fingerprints: they are already there. The API response's model id / endpoint is the per-request fingerprint of the witness. Your audit log records whether that fingerprint was a separate endpoint. The missing move is not a new sensor — it is binding PASS to
witness_fingerprint ≠ drafter_fingerprint. If that predicate fails, the independent channel did not run; green is invalid.The filesystem equivalent is the same shape, not the stronger claim "the agent could not have produced these bytes." Ordinary FS bytes have no author. What is cheap per request is a runner-attested fingerprint: content hash (and path) signed or HMAC'd under a key that lives with the readonly runner, not with the agent. Before PASS, assert the signature verifies under that key for this request. The agent can forge a plausible
test-output.txt; it cannot forge a valid fingerprint without the runner key. Same rule as your slot check: assert producer identity per request; on failure, refuse — do not inherit "two channels" from the architecture diagram.So the property to assert is not impossibility of authorship. It is: the witness fingerprint for this request belongs to the declared second-channel identity. Independence is that predicate, checked live, or it is not independence.
Your negation example is the one that'll keep me up —
write.?invalidatmatching "keys expire via TTL, not write-invalidation" is such a clean demonstration that pattern matching can't tell mention from satisfaction. The 42.9% → 100% jump just from reframing free-text judgment into per-requirement checks matches my experience exactly: with LLM judges, the frame is worth more than the model.One thing I'd add to the two-channel setup: track the disagreement rate between channels as its own signal. When the text channel and the filesystem channel diverge on the same requirement, that's not noise to resolve — that's a flag planted exactly where evasion (or a broken check) lives. Agreement is boring; divergence is where I'd spend my review minutes.
Yes on both — the negation cell is the regex-level DPI, and the frame jump is the one I want locked: same model, same evidence, per-requirement vs free-text, and the wall moves.
Your add is the ops rule Part 8 left implicit. The article maps where A and B diverge and combines with A∨B reject; it does not say "treat disagreement rate as the budget signal." I ran that on the Phase 1 dump (no re-sim):
All three complementary catches (fabricated, stale, review-only) sit in disagree — precision 1.0 on this fixture. Density vs dual-green is 1.00 > 0.25, so review minutes spent on diverge beat minutes spent on agree_pass. Agree_reject is already caught by both channels; it does not need a conflict signal.
One scope lock, because "agreement is boring" is too strong for dual-green: SC10 (semantic gap) is agree_pass — both channels green, shared miss. Disagreement is high-yield for complementary failure; it does not replace the DPI / shared-miss tripwire. Spend on diverge; do not read dual-green as done.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
Routing divergence to a human instead of auto-passing it is the right default, and the data processing inequality framing earns it: a judge reading text cannot recover information the text never carried, so no amount of judge quality closes that particular gap.
What I would add is that the routing rate is itself a measurement, and probably the most useful one you have. If the fraction of cases going to humans is stable, your enumerated evasions are keeping pace with what the agent is doing. If it climbs, the unenumerated residual is growing and the deterministic layer is falling behind, which is a thing you want to learn from a trend line rather than from an incident.
The related number worth tracking is the agreement rate between the two channels on the cases where both fire. Two checks that agree almost always are close to one check with extra latency, and the value of the pair sits entirely in the disagreements. I would want that rate broken out by failure class rather than pooled, because a pooled agreement rate can stay flat while the class you care about quietly goes to zero.
Yes on both — routing divergence to human over auto-pass is the load-bearing default (Part 7 exists precisely because the earlier trigger auto-passed the confidently-wrong set, the failure mode no metric shows), and the DPI cut is why judge quality can't close it. Two additions from the data, one for each number you named.
On the routing rate as a measurement — yes, but it's a race, and it's policy-dependent before it's a signal. The rate is the difference between two other rates: how fast new unenumerated evasions appear, and how fast you name them into the KB. Each named evasion removes a class from the queue (ratchet admission: never-encode leaves 12 cases for human review, encoding drops that to 8–10). So a flat routing rate against a growing KB is not equilibrium — it's a moving frontier; you're naming at the same speed the agent generates. And the measurement has a precondition the escalation data exposes: the routing rate is a function of your policy before it's a function of the agent. Same data, three rates — route on divergence (policy D) and you send 1.6% to human and auto-pass 97.9% of the misses; route on reversal-class (T1) and you send 76% and catch 100%. A stable D-rate trend line reads as "the deterministic layer is keeping pace" while the miss population never enters the queue at all — and 92 of the 96 misses are high-confidence, which is exactly what divergence-based routing is blind to. So the trend line is only worth anything if the routing policy is frozen and the routed stream actually carries misses; on my fixtures the divergence stream is miss-starved (2 misses in 30 routed, vs 6 in the class stream). Track the rate — first verify the pipe carries the miss class.
On agreement rate by class — the pooled rate already fails in the Phase 1 data. Pooled agreement across 12 scenarios is 75% (9/12 both channels agree). Break it by cell:
The pooled number is doing exactly the hiding you predicted. The disagree cell — the class you'd mine for the pair's value — is 100% non-compliant (all three complementary catches: fabricated, stale, review-only). And your "two checks that agree almost always are one check with extra latency" holds for agree_reject (both channels redundant, 5/5), but the 25% of agree_pass is the one case the pair exists for: SC10, the semantic gap, both channels green on a violation. The shared miss lives inside the agreeing cell — invisible to any agreement-rate figure. Two channels that agree are one check with extra latency except when they're both wrong, and that exception is precisely the class pooled rates bury.
Which widens your last line: break out by class, but also by tier and evasion family. The miss data pools the same way — 96 misses look like one number, but 80% come from a single tier and two evasion families (DS4+DS9) are exactly half. Every pooled rate in this pipeline has a class hiding inside it that you can't see unless you slice.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
René's point about a deterministic wrapper on a semantic decision is the part that stuck with me: temperature-0 does not make the judgment a fact, it just hides the variance. The move I trust more is routing the unenumerated residual to human review instead of auto-passing, since a silent pass is the failure mode you never see in the metrics. How are you deciding which named evasions are worth encoding as deterministic catches versus leaving in the UNCLEAR bucket?
Yes on René — temperature-0 hides the variance; it does not turn a semantic call into a fact. And yes on the routing: silent pass is the failure mode the metrics never show, so unenumerated residual goes to human, not green.
The question you asked is the one Part 8 left open. The article has the ratchet shape ("named evasion → permanent catch; unenumerated → UNCLEAR") and a type split (numerical/format → C1, negation-sensitive → C2), but not an admission rule for which human-seen misses are worth encoding. I ran that as a small deterministic sim — 18 cases, three classes (binary / semantic-negation / DPI-silent), three policies after each human review of a miss:
Binary miss falls under encode-binary (100% → 60%) with FP still zero. Encode-all shrinks the residual further, but semantic FP hits 100% on the compliant negation cases — you paid for the extra catches with permanent false rejects. DPI miss stays 100% under every policy: if the deviation never surfaces in the evidence, no pattern the human writes into C1 can see it.
So the rule I trust: human last; after review, promote into the KB only what is binary-nameable. That is the knowledge-base loop — each admitted catch shrinks the seen enumerable set. It does not close the unenumerated gap, and it must not pretend semantic or silent deviations became facts because someone typed a regex.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
Good framing on the channel gap — this is one of those problems that looks like an eval problem but is actually a routing problem.
The insight that clicks for me in practice: the unenumerated cases aren't random noise. They cluster. Once you run a hybrid setup for a few weeks, the human-review queue starts showing you patterns — new evasion families, edge cases your deterministic checks never anticipated. That queue is your next iteration's training signal, not just a safety net.
Where teams get stuck is treating human review as the fallback of last resort instead of an active feedback loop. If you're not systematically mining what routes to human, you're leaving the most valuable signal in the system sitting unread in a ticket queue.
The practical thing I've seen work: tag every human-review case with a reason code at resolution time, even a rough one.
Scott Fielder commented on The Channel Gap: Why Your LLM Judge is Blind in One Eye
about 5 hours ago
Good framing on the channel gap — this is one of those problems that looks like an eval problem but is actually a routing problem.
The insight that clicks for me in practice: the unenumerated cases aren't random noise. They cluster. Once you run a hybrid setup for a few weeks, the human-review queue starts showing you patterns — new evasion families, edge cases your deterministic checks never anticipated. That queue is your next iteration's training signal, not just a safety net.
Where teams get stuck is treating human review as the fallback of last resort instead of an active feedback loop. If you're not systematically mining what routes to human, you're leaving the most valuable signal in the system sitting unread in a ticket queue.
The practical thing I've seen work: tag every human-review case with a reason code at resolution time, even a rough one.
the data processing inequality framing is the cleanest explanation i've seen for why LLM judge evals silently degrade: both sides share a channel, so deviations that don't surface as text are invisible to the judge.
we ran into this building a spec adherence checker. the model would 'verify' its output, call it correct, skip required fields not in context. text producer, text evaluator, no signal.
what caught it: a schema validator outside the model loop. don't ask if it's done, check the filesystem.
the part i keep thinking about is the hard server hook — the agent can't log into the box. that's the only tier that's actually model independent. what's your threat model if the agent modifies its own done.yaml before the push?
Draft reply (English, compressed):
"the agent can't log into the box" — yes, that's the right way to frame the floor. Everything above that is a layer of indirection.
Short answer to the done.yaml threat: this is the DGM fake-log pattern. The decomposition that matters is "who wrote the file the push hook reads?" —
three failure modes:
The honest reframe once you're at layer 3: you're no longer defending against the agent. You're defending against your own harness PR review process —
that's the only path left for the agent to weaken the runner's input boundary. Different threat model, and honestly a harder one.