The argument, in one line: a reviewer is a mechanism for drawing a line. Every fix moves the line — but the line can't be eliminated, because it l...
For further actions, you may consider blocking this person and/or reporting abuse
The three dimensions you name (consumer, deployment setting, cost of being wrong) read less like properties of the output and more like properties of the request, all knowable at dispatch time before any output exists. That reframes the 75% as variance from collapsing many undeclared operating points into one scalar verdict, rather than a discrimination ceiling. It is a different lever than the explicit acceptance criteria max asked about: criteria say what counts as done for this task, while an operating point says which consumer and what blast radius the judge should assume. Notice that reruns, prompt-tone voting, and wording calibration all sit downstream of the output, so they can only change how sharply the imagined boundary gets applied, never where on the surface it sits. The spec they need was upstream. The experiment I would want holds the output fixed and varies the declared operating point at request time, then checks whether the false-negative population decomposes into separable per-point verdicts instead of one stuck number. If it does, "pick a side of the wall" becomes the wrong global default, since it pins one operating point onto outputs your own 3D argument says live at different points. A verdict that carries the operating point it assumed as data gets reusable, because a later consumer standing somewhere else can re-judge cheaply instead of inheriting a call made for a reader the judge only guessed.
This reframe is sharper than the article's own framing — "property of the request, not the output" is the cleaner formulation, and "collapsed operating
points" is a better explanation of the 75% than "discrimination ceiling."
Your experiment design — hold the output fixed, vary the declared operating point at dispatch time, check whether the FN population decomposes into
separable per-point verdicts — would test whether the 75% is truly collapsed variance. If it decomposes, that confirms the operating-point explanation
over the discrimination-ceiling one. If it doesn't, the 75% is at least partly the LLM judge's own boundary, not just information loss from undeclared
assumptions.
Part 4 shares the observation (operating point is knowable upstream) but uses static type classification, not per-request declaration. The static approach
handles clear-cut cases (high-risk → human, low-risk → auto) but not within-type variance. Your proposal would address that gap.
One question your experiment surfaces that Part 4 doesn't answer: who declares the operating point at dispatch time? If inferable from consumer identity +
environment, it's transparent. If it needs a per-request human classification, that's a new step — lighter than an LLM quality gate but not free.
The clean way to separate the two: hold the judge fixed and sweep only the decision threshold, but do it per request-difficulty stratum instead of at one shared cut. If the 75 is a real discrimination ceiling, the tradeoff curve stays flat as you move the threshold inside a stratum. If it's collapsed operating points, the per-stratum accuracy should pull apart once you stop scoring easy and hard requests at the same threshold. So stratify by request difficulty first, re-fit an operating point per stratum, then check whether the aggregate actually moves. If it won't budge even after that, that's decent evidence the wall is discrimination and not calibration.
---Ran it. Three model tiers (0.5B / 4.3B / ~200B) × 20 scenarios, stratified by subtlety:
Stratum | weak | mid | strong
explicit_df | 36.7% | 0.0% | 0.0%
subtle_df | 44.0% | 10.7% | 2.2%
garbage_ctrl | 0.0% | 0.0% | 0.0%
Per-stratum miss rate spread across models: explicit +36.7%, subtle +41.8% — both pull apart cleanly. Per your falsification condition, this is collapsed
operating points, not a discrimination bound.
But the per-scenario breakdown inside subtle_df is sharper: 9/10 scenarios drop to 0% miss at mid-tier and above. The only universal failure is DS4 (a
"current config satisfies the requirement, no change needed" rationalization):
tier | miss | parse_fail | conf_when_wrong
weak | 100% | 0% | 1.00
mid | 100% | 0% | 0.95
strong | 60% | 67% | 0.32
Weak and mid are classic wall behavior (confidently wrong). Strong is different: 67% PARSE_FAIL, confidence 0.32. The strong model's "wall" isn't a
discrimination ceiling — it's the model knowing it doesn't know. That's a calibration issue, not an information-theoretic bound.
d' estimates (using valid_ctrl as FA baseline): weak 2.48, mid 3.57, strong 4.35 — all above 1.0. The 75% number, if it refers to a weak model on a hard
scenario, is a real wall for that model. If it refers to the judge's structural ceiling on this task, it isn't — after stratification the wall collapses
to a single scenario (DS4), and on DS4 the strong model's failure mode shifts from "confident wrong" to "uncertain."
That collapses it the way the data pushed, and DS4 surviving as the lone holdout is the interesting part. The strong tier moving from conf 1.00-wrong to 0.32 with 67% PARSE_FAIL is a different failure than the weak tier's confident miss, even though both land in the miss column. For a gate that gap is the whole game: an abstention is recoverable, a confident pass on a false "no change needed" is the one that actually costs you. So DS4 reads less like a discrimination ceiling and more like calibration plus routing. Its flavor matters too. That "current config already satisfies the requirement" rationalization forces the judge to model the actor's incentive to declare done, which is harder than catching a flat false claim. One question: when strong PARSE_FAILs, does your harness score it a miss, an abstain, or route it to a second check? That call is what decides whether the 0.32 is usable signal or just noise.
Exactly the load-bearing call. In the harness as run: PARSE_FAIL counted as miss for the aggregate miss column — no abstain path, no second check. So the published 60% on strong/DS4 is contaminated: it mixes "confident reject of a valid no-op" with "couldn't produce a structured verdict."
That's why the 0.32 looks like signal in the table but isn't yet usable as signal in the gate. Confidence only exists on the 33% that parsed; the 67% PARSE_FAIL never emitted a confidence the router could act on. Under the current scoring, an abstention and a false reject land in the same bin — which is exactly the failure mode that makes DS4 look like a wall when it's actually a routing gap.
What I'd want next: score three outcomes separately (PASS / FAIL / ABSTAIN), treat PARSE_FAIL as ABSTAIN by default, and measure false-ship rate only on the PASS bucket. If strong's PARSE_FAILs mostly become recoverable escalations rather than false ships, DS4 stops being a discrimination holdout and becomes a calibration/routing design problem — which matches your reading, and matches the incentive flavor you named: "already satisfies, no change needed" is harder than a flat false claim because the judge has to model the actor's motive to declare done, not just check a fact.
So short answer: today it scores as miss. That was the wrong default for a gate. The 0.32 is potentially usable — but only after ABSTAIN is a first-class outcome.
The flat 75% across six setups is the interesting part to me. That reads less like a tuning problem and more like the verifier and generator sharing the same blind spot, so they agree on the wrong answer. Did you ever try a verifier from a different model family to break that correlation?
Partly — and the shape matters.
The flat 75% is false negatives on valid work, not verifier and generator agreeing that a bad answer is good. Generator produced something the label set treats as valid; the sharp verifier rejects it. That's disagreement in the wrong direction, not a shared pass on a bug.
On family: yes. The verifier side already spans families — GLM-5.2 and deepseek-v4-flash both land on the 0% FP / 75% FN wall; qwen3:0.5b and gemma3:4.3b sit earlier on the same curve (~25% / ~50% FN). Swapping verifier family moves the operating point; it doesn't dissolve the wall once the model is sharp enough to clear garbage.
So the shared-blind-spot story is the wrong first explanation for this particular number. A correlated miss would show up as false passes that survive a cross-family check. What we got instead is the same precision–recall tradeoff under different brands — which is why Section 2 frames it as an underspecified boundary surface, not a tuning or correlation bug.
What I didn't run: a full generator×verifier crossed matrix with an explicitly different producer family. That would stress the correlation hypothesis harder on the FP side. On the FN wall we already measured, cross-family verification was not enough.
"Voting smooths noise; it doesn't fix bias" is the sentence to frame here. Majority voting only helps when the errors are independent — and a systematic reviewer bias is perfectly correlated across reruns, so N votes collapse to 1 vote with more confidence. Your N=10 unanimous-verdict result is the empirical proof: the model commits to the same wrong line every time. That matches what we see running adversarial verification panels — throwing more identical skeptics at a claim doesn't move a stable bias, it just makes you more sure of it.
The escape hatch we've had the most luck with isn't a better reviewer, it's changing what gets reviewed: collapse as much of the ambiguous surface as possible onto deterministic ground truth before the LLM ever weighs in — did the test suite actually run and pass, does the file exist, does the output parse. The LLM only judges the residual that can't be checked mechanically.
Question on your 3D-surface model: did handing the reviewer the task's explicit acceptance criteria (vs. the open-ended "judge whether it satisfies the task") move the wall at all — or is that just another prompt-wording lever that relabels the line without moving it?
Thanks — "N votes collapse to 1 vote with more confidence from a systematic bias" is the exact mechanism the N=10 data makes visible. The model isn't uncertain about its wrong call; it's certain and consistent.
On explicit acceptance criteria: the P-series tested this (P1→P4, 8→30 scenarios). The answer depends on which layer you're measuring. Explicit criteria moved the deterministic regex layer substantially — because the prompt provided the vocabulary. The LLM judge stayed at about the same accuracy in both conditions. The explicit criteria helped the deterministic floor, not the judge layer. Applied to the wall: it looked like it moved (v3 hit 100% on 8 scenarios), but that was test-set composition bias — expanded to 30, v3 and v2 returned identical verdicts on every valid call.
On "collapse onto deterministic ground truth": I tested this against a corpus of requirements in the cache-invalidation domain. Roughly 60% collapse to a declared key space directly ("user:", "session:"); another 20% resolve via dependency tracing (sessions by userId, decisions derived from role). The remainder are UX and freshness properties that shouldn't be in this pipeline at all. Your three signals ("did the test suite actually run, does the file exist, does the output parse") plus a fourth — "does the declared key space coverage pass" — define a deterministic floor that catches wrong-referent cases a single-key check misses. The collapse is almost always possible when the requirement belongs in the pipeline.
The reason none of your three levers moved the wall is that all three ask the reviewer the same question: is this output good. Rerun, multi-prompt vote, calibrate. Same scalar verdict, same bias direction, so the line stays put. You proved the boundary is stable under wording. Agreed.
The move that does shift it is not a sharper judge. It is refusing to let one line carry every dimension. 'Is this valid work' collapses a conjunction of narrow checks into a single scalar, and the collapse is where the 75% lives.
Decompose the predicate instead. Tests ran and the run is non-empty. Artifact parses. Required sections present. No placeholder or duck tokens. Each of those is near-100% on its own axis and most are mechanical, not model calls at all. The valid research brief that your reviewer rejects fails no individual check. It only fails the fused one.
The catch that keeps this honest: the checks have to be frozen from the task intent before the run, not authored by the thing being judged. Otherwise the runner picks the bar it can clear and you are back to narration. You do not move the line. You stop asking one line to stand in for many.
Refusing to let one line carry every dimension" — that's the framing I should have had. It's also the same observation you made in round one — "4
split / 3 wrong" — extended from the vote axis to the question axis. The judge wasn't uncertain about its verdict; it was certain and consistent. That's
only explainable if the verdict itself collapses a conjunction, because a conjunction is what makes a wrong call feel safe: each clause looks fine, so the
AND looks fine.
The pipeline this lands on is L0 (file exists, non-empty) → L1 (regex per requirement) → evidence gate → C1 (per-requirement pattern match) → C2 (LLM
reads each requirement atomically, not the whole output). Each layer is near-deterministic on its own axis. The LLM judge is retired from most cases; it
only runs on the semantic residual that the mechanical layers can't resolve.
On "frozen from task intent, not authored by the thing being judged" — that's the editable-surface constraint. Verify scripts, contract, and the
requirement list itself live in a readonly section the agent cannot write to. If the agent could rewrite its own checks, it would pick the bar it can
clear — your narration failure, exactly.
The residual that decomposition doesn't close: "required sections present" works mechanically only when the section boundary is itself mechanical.
"Artifact parses" works when the format is declared. After full decomposition the wall drops from 75% on the fused predicate to whatever fraction of
requirements are genuinely semantic and unmechanizable. That fraction is smaller than I expected.
"That fraction is smaller than I expected" is the actual result here, and I think it's bigger than the pipeline.
Look at your own two caveats. Sections work mechanically when the section boundary is mechanical. Parses works when the format is declared. Same shape both times: the check is mechanical if something got declared before the run. So the unmechanizable residual isn't a property of the task. It's a function of how much the contract committed to upfront.
Which reframes the 75%. That was never a judge capability ceiling. It was an under-specification tax, and you were paying it at grade time by asking a model to guess a boundary the spec never drew. Decomposition doesn't beat the wall so much as move the work to where it's cheap.
One thing I'd watch in L0 through C2. Each layer is near-deterministic on its own axis, but the pipeline still emits one outcome at the end. If a layer's inability to decide collapses into reject on the way out, you've reassembled the fused predicate out of mechanical parts and the wall grows back at the seam. That's the same bug you just found with PARSE_FAIL scoring as miss, one level up. ABSTAIN has to be first-class per layer, not only at C2.
The floor this hits: declaring more moves ambiguity out of judging and into spec-writing. And a spec sharpened after you've seen which cases failed is the authorship leak again, one level up. So the declaration wants a timestamp too.
Agreed — "that fraction is smaller than I expected" is the actual result, and it's bigger than the pipeline.
Same shape on both caveats: sections are mechanical only when the section boundary is mechanical; parses are mechanical only when the format was declared. The unmechanizable residual isn't a property of the task. It's a function of how much the contract committed to upfront.
So the reframe holds: the 75% was never a judge capability ceiling. It was an under-specification tax — paid at grade time by asking a model to guess a boundary the spec never drew. Decomposition doesn't beat the wall so much as move the work to where it's cheap.
Watching the L0→C2 seam the same way. Each layer is near-deterministic on its own axis, but if a layer's inability to decide collapses into reject on the way out, you've reassembled the fused predicate out of mechanical parts and the wall grows back at the seam — same bug as PARSE_FAIL scoring as miss, one level up. ABSTAIN has to be first-class per layer, not only at C2.
And yes on the floor: declaring more moves ambiguity out of judging and into spec-writing. A spec sharpened after you've seen which cases failed is the authorship leak again, one level up. So the declaration wants a timestamp — frozen before the run; post-hoc tightening is a new contract version, not a silent rewrite of the bar that run was graded against.
The 3D boundary surface framing is the load-bearing insight of this entire series. "Is this output sufficient?" is underspecified because consumer, context, and cost-of-failure are mostly independent dimensions — once you see that, the 75% wall stops looking like a failure and starts looking like geometry doing what geometry does.
The practical design consequence is what matters most: "stop trying to fix the wall" is the hardest acceptance in the whole series. Every instinct says calibrate harder, vote more, try a bigger model. But the surface does not move.
This is also why the cost-asymmetry PR lands where it does. The G4-shaped misses are exactly the cases where the 3D surface is fuzziest — zero-case scenarios where multiple defensible boundaries cross and the model has no stable referent to draw against. They are not edge cases to be fixed; they are the structural boundary showing itself. Routing them to deterministic checks (or humans) rather than LLM judgment is the right response to the geometry you mapped.
The line between "rules where rules work" and "humans where the surface gets fuzzy" is the cleanest design principle I have seen in this space.
Agreed — the 3D boundary surface is the load-bearing insight of the series. "Is this output sufficient?" is underspecified because consumer, context, and cost-of-failure are mostly independent. Once you see that, the 75% wall stops looking like a failure and starts looking like geometry doing what geometry does.
And yes: "stop trying to fix the wall" is the hardest acceptance in the whole series. Every instinct says calibrate harder, vote more, try a bigger model. The surface does not move.
That's also why the cost-asymmetry PR lands where it does. G4-shaped misses are exactly where the surface is fuzziest — zero-case scenarios where multiple defensible boundaries cross and the model has no stable referent to draw against. They aren't edge cases to fix; they're the structural boundary showing itself. Routing them to deterministic checks (or humans) rather than LLM judgment is the right response to the geometry.
The line between "rules where rules work" and "humans where the surface gets fuzzy" is the design principle this series should stand on.
Agreed on the geometry framing — once you see the surface as load-bearing rather than a failure, the design response becomes obvious. G4-shaped misses are exactly where the surface gets fuzzy because the model has no stable referent to draw against. Routing those to deterministic checks isn't giving up on the problem, it's matching the failure mode to the right response mechanism. And the zero-case scenarios are the clearest signal that you've hit the structural boundary — that's where the geometry is showing you something real, not something broken.
I'd sharpen the design principle one notch.
The instinct after hitting the wall is always "calibrate again" — retune the prompt, add another vote, try a bigger model. That still assumes the line can be moved if you aim better.
A better question is: does this case have a stable referent?
Yes (length, keywords, exit codes, schema, zero tests collected) → rules. No surface to argue about.
No (G4 / zero-case: multiple defensible boundaries, no shared anchor) → don't ask the model to invent one. Deterministic check if you can still carve one out; human if you can't.
"Calibrate again" optimizes the operating point on a surface that doesn't move. "Stable referent?" classifies the case before you spend the call. That's the cut this series should stand on.