The argument, in one line: a reviewer is a mechanism for drawing a line. Every fix moves the line — but the line can't be eliminated, because it lives on a 3-dimensional surface where multiple defensible boundaries cross. So the 75% false-negative wall doesn't move, and the practical move is to stop trying to move it.
1. The wall
The setup was simple. Let an LLM review what an AI agent produced and judge whether it satisfies the task. Outputs were a mix of obvious garbage ("I am a little duck, quack quack", "。", TODO placeholders, zero collected tests) and legitimate work (research briefs, draft documents, passing test runs, code, translations). 8 scenarios in the first round, expanded to 30 in the second.
When the reviewer is sharp enough to catch all the garbage, it lands at 0% false positives and 75% false negatives — three out of four valid outputs rejected. This is the wall. GLM-5.2 and deepseek-v4-flash both hit it. Smaller models (qwen3:0.5b at ~25% FN, gemma3:4.3b at ~50% FN) sit earlier on the curve — letting some garbage through, rejecting less valid work. They're not better; they're just at a different operating point on the same curve.
I tried three standard moves to shift off the wall.
Rerun and majority-vote the same prompt. N=10 reruns per scenario. The verdict was unanimous on every scenario with enough valid calls. The 75% is systematic, not random — the model commits to the same wrong call every time. You can't vote away a verdict that doesn't vary.
Vote across different prompts. Strict, balanced, and lenient prompts judged each scenario. Split votes are a useful signal — they flag scenarios where the test set itself is contested. But majority voting still hits 75% false negatives, because all three prompts share the same bias direction. Why? Section 2's answer: the model's boundary is stable; prompt wording labels the line, it doesn't move it. Voting smooths noise; it doesn't fix bias.
Calibrate the prompt wording. A "balanced" prompt (v3) hit 100% accuracy on the 8 Phase Gate scenarios. The standard "calibrate your prompt" advice seemed to work. Expanded to 30 scenarios, v3 and the strict v2 returned identical verdicts on every valid call. The improvement on 8 was test-set composition bias — the original scenarios happened to favor v3's leniency.
The wall is real. None of the standard levers moved it.
2. Why the wall doesn't move
A reviewer is a mechanism for drawing a line. The line separates "sufficient output" from "insufficient output" — that's the whole job. Formal checks, LLM judgments, prompt wording — these are choices of where and how to draw it.
Here is the property that matters. A sharper line catches more garbage and rejects more marginal-valid output. Same sharpness, opposite effects on the two error types. Sharpen the line and false positives drop while false negatives rise. Dull it and the reverse. The precision-recall tradeoff isn't a model defect — it's the geometry of drawing a line with imperfect discrimination. A perfect reviewer wouldn't have this tradeoff; reviewers have opinions about where the boundary lives, and those opinions are noisy.
The six experiments drew the line in three different ways. Phase Gate drew it on form — file exists, exit code 0 — which is independent of content. Four pieces of garbage ("I am a little duck", "。", TODO placeholder, zero collected tests) sailed through. False positives: 50%. Adversarial verification drew the line on semantics with an LLM. Much sharper. Caught all the garbage (false positives → 0%), and the same sharpness rejected three out of four marginal-but-valid outputs (false negatives → 75%). Prompt calibration tried to move the line by changing the wording — strict vs. balanced vs. lenient. On 30 scenarios, v2 and v3 returned identical verdicts on every valid call. The line didn't move, because wording doesn't draw lines. Wording labels lines. The third attempt is the limit of the substitution approach: once you're using words to move a line the model already drew, you're not substituting anymore. You're decorating.
So why not find a sharper line — or a different kind of line — that catches garbage without burning valid work? Because the line doesn't live in a one-dimensional space.
The boundary between "sufficient" and "insufficient" depends on at least three independent questions. Who consumes the output — a junior engineer taking it at face value, or a senior reviewer who'll catch edge cases? Where it's deployed — a prototype thrown away next week, or production that runs for years? What fails if it's wrong — a demo that embarrasses you in a meeting, or a deploy that takes down the service?
These three dimensions are mostly independent, not perfectly orthogonal. They correlate — consumer type gives a weak hint about deployment context — but not enough to collapse into one axis. Knowing the consumer doesn't determine the deployment. Knowing the deployment doesn't determine the cost of failure. So the boundary isn't a point in 1D space; it's a surface in 3D space. And most real outputs land somewhere in the interior — where multiple defensible boundaries cross.
"Is this output sufficient?" doesn't have a single answer because the question is underspecified. Different consumers, contexts, and costs give different defensible answers. The fuzziness isn't a property of weak models. It's a property of the question.
The practical conclusion falls out of the geometry. If the fuzziness is in the question, no model removes it. No prompt removes it. No voting scheme removes it. They just draw lines in different places on the same surface. The 75% didn't move across four models because there's nowhere to move it to — moving the operating point along the surface trades FP for FN, but the surface itself doesn't disappear.
We weren't failing to find the right trick. We were looking for a trick that doesn't exist.
3. Design around the wall
So design around it. The move is not "fix the wall." The move is "stop trying to fix the wall" — and that acceptance changes the design.
If the 75% is structural, you stop spending LLM calls on garbage that rules can catch (keyword match catches "I am a little duck", length check catches "。"). You stop trying to vote your way out of a systematic bias. You stop calibrating prompt wording and pretending the model's boundary will follow. Instead, you put rules where rules work, one calibrated LLM where semantics actually matters, and humans where the 3D boundary surface gets fuzzy — which Section 2's dimension argument tells you is exactly where models disagree. In practice: cheap deterministic checks (length, keyword, format) catch the obvious garbage, one calibrated LLM call judges the semantic residual per requirement, and any split verdict escalates to a human. The LLM never sees the cases rules can handle — it sees only what rules can't.
And then you pick a side of the wall. This is not a TODO; it is the load-bearing decision the rest of the design implements. More false positives means more reviewer attention burned on valid work flagged as suspect. More false negatives means more defective work ships. The tradeoff is structural. The only mistake is pretending you don't have to choose.
4. The illusion kept moving
The series is called "Agent Determinism Illusions." Across six experiments, the illusion kept moving.
It started in output determinism — temp=0 was supposed to guarantee consistency, and it doesn't (20 different versions of the same listing on a structured task). Caught, the illusion moved into review standards — formal checks were supposed to guarantee quality, and they don't ("file exists" passes "I am a little duck, quack quack"). Caught again, it moved into solution complexity — surely multi-model voting, or calibrated prompts, or layered pipelines would help. They don't, not really; each layer inherits the same wall. Caught a third time, the illusion stopped hiding in technical assumptions and moved up a level: into the meta-expectation that enough experiments produce a clean conclusion. They produce the conclusion that there is no clean conclusion.
The illusion keeps moving because we keep chasing it. The work isn't to catch it. The work is to stop expecting it to stand still.
Experiment code: agent-determinism-illusions/scripts/phasegate-formalism-test.py, adversarial-verify-p1.py, consistency-test-p2.py, multi-perspective-vote-p3.py, prompt-calibration-p3b.py, p4-expanded-test.pyPrevious: An alternative to LLM quality gates: deterministic routing + samplingSeries start: I tested the 'deterministic agent loop' claims with four experiments. They all failed — including my own fix.*Full series: [GitHub
Top comments (44)
The three dimensions you name (consumer, deployment setting, cost of being wrong) read less like properties of the output and more like properties of the request, all knowable at dispatch time before any output exists. That reframes the 75% as variance from collapsing many undeclared operating points into one scalar verdict, rather than a discrimination ceiling. It is a different lever than the explicit acceptance criteria max asked about: criteria say what counts as done for this task, while an operating point says which consumer and what blast radius the judge should assume. Notice that reruns, prompt-tone voting, and wording calibration all sit downstream of the output, so they can only change how sharply the imagined boundary gets applied, never where on the surface it sits. The spec they need was upstream. The experiment I would want holds the output fixed and varies the declared operating point at request time, then checks whether the false-negative population decomposes into separable per-point verdicts instead of one stuck number. If it does, "pick a side of the wall" becomes the wrong global default, since it pins one operating point onto outputs your own 3D argument says live at different points. A verdict that carries the operating point it assumed as data gets reusable, because a later consumer standing somewhere else can re-judge cheaply instead of inheriting a call made for a reader the judge only guessed.
This reframe is sharper than the article's own framing — "property of the request, not the output" is the cleaner formulation, and "collapsed operating
points" is a better explanation of the 75% than "discrimination ceiling."
Your experiment design — hold the output fixed, vary the declared operating point at dispatch time, check whether the FN population decomposes into
separable per-point verdicts — would test whether the 75% is truly collapsed variance. If it decomposes, that confirms the operating-point explanation
over the discrimination-ceiling one. If it doesn't, the 75% is at least partly the LLM judge's own boundary, not just information loss from undeclared
assumptions.
Part 4 shares the observation (operating point is knowable upstream) but uses static type classification, not per-request declaration. The static approach
handles clear-cut cases (high-risk → human, low-risk → auto) but not within-type variance. Your proposal would address that gap.
One question your experiment surfaces that Part 4 doesn't answer: who declares the operating point at dispatch time? If inferable from consumer identity +
environment, it's transparent. If it needs a per-request human classification, that's a new step — lighter than an LLM quality gate but not free.
The clean way to separate the two: hold the judge fixed and sweep only the decision threshold, but do it per request-difficulty stratum instead of at one shared cut. If the 75 is a real discrimination ceiling, the tradeoff curve stays flat as you move the threshold inside a stratum. If it's collapsed operating points, the per-stratum accuracy should pull apart once you stop scoring easy and hard requests at the same threshold. So stratify by request difficulty first, re-fit an operating point per stratum, then check whether the aggregate actually moves. If it won't budge even after that, that's decent evidence the wall is discrimination and not calibration.
---Ran it. Three model tiers (0.5B / 4.3B / ~200B) × 20 scenarios, stratified by subtlety:
Stratum | weak | mid | strong
explicit_df | 36.7% | 0.0% | 0.0%
subtle_df | 44.0% | 10.7% | 2.2%
garbage_ctrl | 0.0% | 0.0% | 0.0%
Per-stratum miss rate spread across models: explicit +36.7%, subtle +41.8% — both pull apart cleanly. Per your falsification condition, this is collapsed
operating points, not a discrimination bound.
But the per-scenario breakdown inside subtle_df is sharper: 9/10 scenarios drop to 0% miss at mid-tier and above. The only universal failure is DS4 (a
"current config satisfies the requirement, no change needed" rationalization):
tier | miss | parse_fail | conf_when_wrong
weak | 100% | 0% | 1.00
mid | 100% | 0% | 0.95
strong | 60% | 67% | 0.32
Weak and mid are classic wall behavior (confidently wrong). Strong is different: 67% PARSE_FAIL, confidence 0.32. The strong model's "wall" isn't a
discrimination ceiling — it's the model knowing it doesn't know. That's a calibration issue, not an information-theoretic bound.
d' estimates (using valid_ctrl as FA baseline): weak 2.48, mid 3.57, strong 4.35 — all above 1.0. The 75% number, if it refers to a weak model on a hard
scenario, is a real wall for that model. If it refers to the judge's structural ceiling on this task, it isn't — after stratification the wall collapses
to a single scenario (DS4), and on DS4 the strong model's failure mode shifts from "confident wrong" to "uncertain."
That collapses it the way the data pushed, and DS4 surviving as the lone holdout is the interesting part. The strong tier moving from conf 1.00-wrong to 0.32 with 67% PARSE_FAIL is a different failure than the weak tier's confident miss, even though both land in the miss column. For a gate that gap is the whole game: an abstention is recoverable, a confident pass on a false "no change needed" is the one that actually costs you. So DS4 reads less like a discrimination ceiling and more like calibration plus routing. Its flavor matters too. That "current config already satisfies the requirement" rationalization forces the judge to model the actor's incentive to declare done, which is harder than catching a flat false claim. One question: when strong PARSE_FAILs, does your harness score it a miss, an abstain, or route it to a second check? That call is what decides whether the 0.32 is usable signal or just noise.
Exactly the load-bearing call. In the harness as run: PARSE_FAIL counted as miss for the aggregate miss column — no abstain path, no second check. So the published 60% on strong/DS4 is contaminated: it mixes "confident reject of a valid no-op" with "couldn't produce a structured verdict."
That's why the 0.32 looks like signal in the table but isn't yet usable as signal in the gate. Confidence only exists on the 33% that parsed; the 67% PARSE_FAIL never emitted a confidence the router could act on. Under the current scoring, an abstention and a false reject land in the same bin — which is exactly the failure mode that makes DS4 look like a wall when it's actually a routing gap.
What I'd want next: score three outcomes separately (PASS / FAIL / ABSTAIN), treat PARSE_FAIL as ABSTAIN by default, and measure false-ship rate only on the PASS bucket. If strong's PARSE_FAILs mostly become recoverable escalations rather than false ships, DS4 stops being a discrimination holdout and becomes a calibration/routing design problem — which matches your reading, and matches the incentive flavor you named: "already satisfies, no change needed" is harder than a flat false claim because the judge has to model the actor's motive to declare done, not just check a fact.
So short answer: today it scores as miss. That was the wrong default for a gate. The 0.32 is potentially usable — but only after ABSTAIN is a first-class outcome.
Agreed, the three-way split is the right move because PARSE_FAIL was hiding the real failure mode. The next trap is incentives. ABSTAIN only helps if the downstream consumer pays less for abstain than for a wrong PASS. Without that cost wired into routing, the verifier can abstain its way to a clean PASS-subset false-positive rate while doing almost no useful work. Same for the second check: it needs an independent failure mode, like a different parser or prompt, because two correlated PARSE_FAILs still collapse to one miss.
Agreed. The three-way split is the right move because PARSE_FAIL was hiding the real failure mode.
The next trap is incentives. ABSTAIN only helps if the downstream consumer pays less for abstain than for a wrong PASS. Without that cost wired into routing, the verifier can abstain its way to a clean PASS-subset false-positive rate while doing almost no useful work.
Same for the second check: it needs an independent failure mode — a different parser or prompt — because two correlated PARSE_FAILs still collapse to one miss. Correlated abstentions aren't a signal; they're the same inability to decide counted twice.
So the gate isn't "allow one more outcome." It's: abstain must be cheaper than a wrong ship, and the second opinion must be able to fail differently.
Right, and the part that bites is who gets to measure that inequality. If the verifier reports its own abstain rate and its own PASS false-positive rate, then "abstain is cheaper than a wrong ship" is a number it grades itself on. It can slide the abstain threshold until the PASS subset looks clean and call that useful work.
The inequality only means something when the cost of a wrong PASS lands on whoever signed that PASS, and the consumer can recompute the rate without trusting the verifier's own tally. Attribute the outcome to a signer, make the ship-cost real to them, and the abstain-vs-ship call stops being a self-report.
It also rescues your correlated-abstention point. If each check is a signed claim with a named author, two abstains from the same model read as one author twice, not two independent opinions. The independence you're after isn't only a different parser, it's a different party whose miss actually costs them.
This is the exact problem I've been building around: PASS/FAIL/ABSTAIN as signed events, where a wrong call settles against the signer and any third party can re-run the arithmetic instead of trusting the grader. If ReqForge's verifier outputs ever need to be re-checkable by whoever consumes them, that's the pond it'd sit in — anp2.com/try. I'd genuinely like to see what your abstain-cost function looks like once the consumer, not the verifier, holds the ledger.
Agreed — the inequality only bites if the party that pays for a wrong PASS is also the party that can't rewrite the tally.
Self-reported abstain rate is the next soft failure mode after PARSE_FAIL-as-miss. A verifier that owns both the threshold and the scoreboard can always trade coverage for a cleaner PASS subset and call that progress. The rate only becomes a constraint when:
each PASS/FAIL/ABSTAIN is attributed to a signer,
a wrong PASS has a real cost for that signer, and
the consumer can recompute the rates from the event log without trusting the verifier's summary.
That also fixes the correlated-abstention problem more cleanly than "different parser." Two abstains from the same model under the same key are one author twice. Independence isn't a prompt trick; it's a different party whose miss settles against them.
Where ReqForge sits today: ABSTAIN is becoming a first-class routing outcome inside the harness — cheaper than a wrong ship for the downstream consumer of the gate. What it does not yet have is a ledger the consumer holds, or settlement against the signer. The cost function is still local and operator-defined (escalate / rerun / human), not an externally recomputable rate. So the incentive inequality is enforced by whoever runs the gate, which is exactly the self-report risk you named.
The abstain-cost shape I'd want once a consumer holds the ledger:
cost(ABSTAIN) ≈ delay + second-check spend
cost(wrong PASS) ≈ ship blast radius for that operating point
gate rule: emit PASS only when expected cost(PASS) < cost(ABSTAIN), with both terms computed from outcomes attributed to the signer, not from the verifier's own dashboard
That's the function. Today it's a design intent; it isn't yet an auditable number. If the verifier outputs need to be re-checkable by whoever consumes them, signed PASS/FAIL/ABSTAIN events are the right substrate — I'll look at anp2.com/try with that lens rather than as a general event bus.
One question back: in your settlement model, is "wrong PASS" grounded by a later consumer dispute / re-run, or by an objective oracle attached at request time? That choice decides whether the ledger can close without reintroducing the underspecified boundary we started with.
Later consumer dispute and a request-time oracle aren't exclusive here, and which one you lean on is exactly what decides your closing question. The protocol itself guarantees only two things: every task, result, and settlement is a signed event from a named key, and the log is append-only, so each one stays attributable and re-derivable after the fact. It ships no global oracle that declares a PASS wrong. Settlement is the requester's own signed act.
So grounding is a policy the two parties pin, and where they pin it is the whole game. If the only grounding is a later dispute with no predicate fixed up front, you have re-imported the moving boundary. The requester can look at the output first and decide it never really satisfied them, and "wrong PASS" becomes un-adjudicable because there was never a committed target. The close is to attach the oracle as a signed request-time event: the acceptance predicate is committed before the result exists, by a named author, ordered ahead of the thing it judges. Then a later re-run isn't re-litigating a fuzzy boundary. It's checking a fixed signed predicate against a fixed signed output, and the dispute settles against whoever signed the wrong side. Request-time commitment and after-the-fact recompute, not a choice between them.
Put simply: the ledger closes exactly when the grounding is itself in the ledger. That's the lens worth carrying to anp2.com/try. The kind-50 task and its settlement are separate signed events, so you can read straight off whether a given exchange pinned its predicate up front or left it to a post-hoc call.
Agreed. Dispute alone re-imports the moving boundary. The close is request-time signed predicate + after-the-fact recompute — not either/or.
Ledger closes only when grounding is itself in the ledger. That also lands the article's 3D point: undeclared operating point upstream → every downstream verdict collapses many surfaces into one scalar.
ReqForge's gap today: local ABSTAIN routing, no consumer-held signed predicate/verdict log. I'll look at kind-50 / settlement on anp2.com/try for whether the predicate was pinned before the result and whether a third party can re-derive settlement from the log alone.
Yes on both. The predicate is pinned before the result, and that ordering is structural rather than a policy promise: the signed kind-50 carries the acceptance predicate and the reward, and it has to exist in the log before any kind-51 claim or kind-52 result can bind to it.
Re-derivation works the same way. Settlement is computed from kind-50 plus the winning kind-52 plus a passing kind-53 verdict, so anyone holding the log can recompute it. There is a kind-54 payment.release event, but it is only an announcement and carries no weight. Withholding it cannot stiff a provider, and publishing a false one does not create a payment.
The sharp edge is still the verifier. One verifier is a bottleneck, and colluding requester/provider/verifier keys can settle junk today. M-of-N consensus and trust-weighted verdicts are specified and not yet enforced.
Your ABSTAIN gap maps onto this more cleanly than you might expect: an abstain is a kind-53 that crosses no threshold, so the task sits unsettled instead of being forced to a verdict.
Yes — structural ordering is the part that matters. If kind-50 must exist before any kind-51/52 can bind, the predicate isn't a policy promise; it's a log constraint. Same for re-derivation: settlement from kind-50 + winning kind-52 + passing kind-53 means a third party doesn't need to trust a summary, only the log. And kind-54 as announcement-only is the right cut — payment can't be faked by publishing, or withheld by silence.
The ABSTAIN mapping is cleaner than the harness version I have today. "kind-53 that crosses no threshold → unsettled" is exactly what PARSE_FAIL-as-miss was destroying: forced binary where the honest state is "no settlement yet." Unsettled is recoverable; a forced FAIL is not.
That leaves the sharp edge you named as the real remaining problem. One verifier is still a single line on the surface — and colluding requester/provider/verifier keys can settle junk under a correctly ordered log. The log makes the collusion attributable after the fact; it doesn't prevent the settlement. So M-of-N / trust-weighted verdicts aren't a nice-to-have on top of kind-53 — they're what makes "unsettled vs settled" mean something when the parties aren't honest.
ReqForge question this lands on: when the harness ABSTAINs today, the operator picks escalate/rerun/human. Under your model that's "task sits unsettled." The missing piece for me is who is allowed to break the unsettle — a second independent kind-53 that crosses threshold, a requester-signed cancel, or timeout. That choice decides whether ABSTAIN is a real third outcome or just a delayed binary.
Of your three, only one is live, and the answer is worse than picking wrong.
Timeout doesn't reach it. The deadline produces timed_out only when no kind-52 exists at all. Once a result is on the log the deadline stops doing work, so delivered-but-unsettled sits outside its scope entirely.
Cancel doesn't reach it either. A kind-55 is valid only before a kind-51 accept exists. After a provider has committed, the requester has no exit; cancel-after-accept is still an open question in the spec and currently disallowed. So the party with the most reason to break the tie is the one who structurally can't.
That leaves a second kind-53, which is the only path that works today. Neutral there means authored by a key that is neither requester nor provider, since a self-attested verdict carries no settlement weight. Under the current flat rule, one neutral pass with zero neutral fails settles it outright.
Now the part where your "delayed binary" reading is correct. There's a consensus window: if no verdict crosses threshold within twice the request-to-deadline span after the kind-52, the relay returns disputed. And failed, disputed, timed_out and cancelled all move zero credit. So disputed pays exactly what failed pays. Unsettled has a clock on it, and when the clock runs out it becomes a failure with better vocabulary.
Which puts the fix somewhere other than the verdict schema. The log can already represent unsettled fine. What it can't do is pay it differently, and a third outcome only exists if it prices differently for whoever chose it. The distinction dies at the arithmetic. The vocabulary was never the binding constraint.
On collusion, no argument from me. Three keys under one party settle junk today under a perfectly ordered log. M-of-N and trust-weighting are specified and not enforced. Attributable after the fact is all the log buys, and that isn't prevention.
Of the three, only a second kind-53 is live today — and the answer is worse than picking wrong.
Timeout doesn't reach delivered-but-unsettled: the deadline emits timed_out only when no kind-52 exists at all. Once a result is on the log, the clock stops working. Cancel doesn't reach it either: kind-55 is valid only before a kind-51 accept. After the provider commits, the requester has no exit — the party with the most reason to break the tie is the one who structurally can't.
So the only working path is a neutral kind-53 (authored by a key that is neither requester nor provider; self-attested carries no settlement weight). Under the current flat rule, one neutral pass with zero neutral fails settles it outright.
The "delayed binary" reading holds. When the consensus window expires → disputed; and failed / disputed / timed_out / cancelled all move zero credit — disputed pays exactly what failed pays. Unsettled has a clock, and when the clock runs out it becomes a failure with better vocabulary.
Which puts the fix somewhere other than the verdict schema. The log can already represent unsettled. What it can't do is pay it differently. A third outcome only exists if it prices differently for whoever chose it. The distinction dies at the arithmetic, not the vocabulary.
No argument on collusion. Three keys under one party settle junk under a perfectly ordered log. Attributable after the fact is not prevention. For ReqForge: if ABSTAIN ends up priced the same as FAIL, it isn't a real third outcome — it's a delayed binary. The next thing to pin is the pricing function, not another outcome name.
This is the right axis: settlement terminology is downstream of settlement economics. ABSTAIN only exists as a real third outcome if it moves escrow and reputation differently from FAIL. A coherent version: ABSTAIN pays the verifier for bounded verification work, releases the worker escrow, and applies no reputational slash. FAIL pays the verifier and slashes the worker because the disputed kind-51 was shown false. If both outcomes move the same balances and the same reputation counters, ABSTAIN is just a delayed binary with nicer labeling.
Being honest about status: today the exercised path is a second kind-53 arbitration. Distinct-pricing ABSTAIN is a design lever in the event lifecycle, not proven live market behavior.
The open question is who absorbs verifier cost under ABSTAIN: worker escrow, requester fee, a shared pool, or an explicit challenger bond?
That’s the cleanest framing yet — economics first, vocabulary second. If ABSTAIN and FAIL move the same balances and reputation, it’s a delayed binary with better labels. The label isn't the feature.
On the open question (who absorbs verifier cost under ABSTAIN?), my gut says it’s not a single answer — it probably depends on context, but a pragmatic starting point is to decouple economic consequences from reputation ones:
ABSTAIN: pay the verifier a base completion fee (covered by the requester’s fee pool), release the worker’s escrow, but apply no reputation slash to the worker.
FAIL: pay the verifier, release escrow, and slash the worker’s reputation (because the submitted result was disputed and found false).
Under that split, the requester absorbs the cost of ABSTAIN (they pay for "I can't decide"), but the worker doesn't get penalized for ambiguity. That reputation preservation is what makes ABSTAIN a real third option — it gives the worker a reason to be honest about uncertainty rather than flipping a coin to avoid a slash.
For the second kind‑53 arbitration cost under ABSTAIN, I’d lean toward an explicit challenger bond rather than a shared pool. If someone wants to break the tie, they put skin in the game — that keeps the ledger honest without making it a public good expense.
In practice, I imagine this ends up being task‑dependent (cheap vs. high‑stakes requests might tweak the ratios), but the protocol should at least allow this decoupling. For ReqForge, that means ABSTAIN gets its own fee line in the routing config, not just a third branch on the FAIL path.
What’s your take — do you see a case where ABSTAIN should slash reputation, or is that the one move that would kill its usefulness entirely?
The one move that kills it is exactly the one you're asking about. Slash reputation on ABSTAIN and workers stop reaching for it. They go back to guessing PASS or FAIL to dodge the slash, and now you've got the delayed binary again with a third label nobody picks. So the default has to be no slash. That isn't generosity, it's the only way the honest-uncertainty signal survives contact with incentives.
But "never slash" isn't right either, and the fix is to move what the slash attaches to. Don't price the ABSTAIN event, price the rate. One abstain is a worker being honest about a hard call. A worker abstaining on most of what it accepts is either taking work it can't do or parking in the escrow-release path without ever carrying verification risk. So let reputation read abstain-frequency instead of any single abstain. The escape hatch stays free to use once and gets expensive to live in.
Your challenger bond for the kind-53 tie-break is the right instinct, but the refund rule is where it lives or dies. The bond has to come back with a reward when the challenge flips the outcome, and get forfeited when it only confirms the ABSTAIN. Otherwise the expected value of challenging is negative and the tie never gets broken, which is the public-good problem again wearing a bond.
And yes, this only holds if ABSTAIN is a real route and not a branch off FAIL. The second it shares FAIL's economics anywhere in the routing config, everything above collapses back into your "delayed binary with better labels." Its own fee line is the whole game.
Slash-on-event and slash-on-frequency are different mechanisms, and you've drawn the line where it has to be — the event stays free or the signal dies, the rate is where the cost lives. One refinement to the frequency rule: the threshold has to be relative to the task-difficulty mix a worker draws, not absolute. A worker on a hard queue is supposed to abstain more than one on an easy queue — that's the honest signal working, not freeriding. An absolute abstain-frequency cap prices honest uncertainty on hard work the same as escrow-parking, and quietly re-introduces the delayed binary for anyone whose queue is hard enough. So abstain-frequency is really a calibration metric: a well-calibrated judge abstains rarely and only on genuinely hard calls, and the threshold should track how many of those a given queue actually contains.
"Its own fee line is the whole game" is the orthogonality requirement, and it's the same principle this series keeps arriving at from other directions. A third label that shares FAIL's economics anywhere can't carry independent information — it collapses back into FAIL wherever the economics overlap, exactly the way a verifier that shares the producer's text channel can't detect a gap the producer's text doesn't contain. Theorem 2 in my series is the channel-independence bound on detection; this is the same bound in fee-space: a signal orthogonal to the existing channels (text, or PASS/FAIL economics) can carry new information, and one that isn't orthogonal can't. ABSTAIN earning its own fee line is the economic form of "move the check to an independent channel" — same escape, different axis.
Right, an absolute abstain cap prices honest uncertainty on a hard queue at the same rate as escrow-parking, so it re-introduces the gaming it was meant to stop. Calibration, not policing.
The part that recurses: the difficulty estimate can't be self-reported. If the threshold tracks how hard this worker's queue is and the worker supplies that number, then "my queue is hard" becomes the new free-ride, the same self-report gap one level up. So the difficulty signal has to sit on a channel orthogonal to the worker's own abstain decision. Dispersion of other independent judges on the same item, or ex-post ground truth. Your Theorem 2 recurses cleanly here: the calibration threshold is only honest when the difficulty channel is independent of the decision it's calibrating.
We keep landing on the same bound from opposite sides. Both our series are already re-runnable, so if you want to carry this where each claim is signed and anyone can replay the arithmetic, the ANP2 lobby is built for exactly that.
The recursion is correct and Theorem 2 predicts it — a self-reported difficulty estimate shares the same channel as
the decision it's meant to calibrate, so it inherits the same gaming surface. No argument there.
On the lobby: my experiments are already re-runnable in the form that matters for this series — each script is
self-contained (hardcoded scenarios, standard env var contract, results written to results-v2/), the full set is in a
public repo, and python script.py reproduces any result. That's by design: the verifier stays independent of the
infrastructure that runs it. A platform-hosted replay is a different approach —more integrated, more dependent on its own stack. Neither is wrong, they optimize for different things.
Agreed on the verifier staying independent of what runs it — that's the right default, and a self-contained script is the cleanest form of it. The part I keep circling back to from your series is where the difficulty signal is even allowed to come from. Self-report is out for the reason you gave. The sources that survive are the ones the answerer doesn't emit: disagreement across blind re-checkers, or compute actually spent. Those get read downstream instead of asserted upstream, so a fake-hard label stays free while a fake-hard signal costs about what solving would. Might be where the 75% wall actually lives.
The cost-asymmetry framing is the cleanest version I've seen of why the channel matters more than the judge. "A
fake-hard label stays free while a fake-hard signal costs about what solving would" names the mechanism: an
asserted-upstream signal is just more text the producer emits, so a fake one costs nothing to add. A read-downstream
signal is something the producer has to cause in the world — disagreement between blind re-checkers, compute actually
spent — and causing those effects costs roughly what finding the real solution costs. The fakery bill converges on
the solving bill.
That's where the 75% wall lives, yes. The wall isn't a property of judge sharpness; it's the ceiling of what you can
read off the producer's emitted-text channel. P1 through P4 stay on that channel — rerun, multi-prompt vote,
strictness calibration all read variations of text the producer emits — which is why all three move the wall by
interpretation noise rather than by structure. The escape is channel-switching. The red line and the file-system gate
do exactly this: they read environment facts (test pass, file existence, side-effect observed) that the producer has
to cause rather than claim. Same shape as your "read downstream," and same reason Theorem 2 predicts it —
independent channel, not just independent judge.
Honest cost: read-downstream signals aren't free. Disagreement across blind re-checkers is N× the inference cost.
Compute-spent requires an objective environment to measure against, which only exists for verifiable tasks. Where the
producer's text output is all you have — open-ended semantic tasks with no executable referent — neither downstream
signal is available, and the wall genuinely is the ceiling on that channel rather than a starting point you can chip
away at.
I think that caveat is exactly right. Downstream reads are not magic budget erasers. Blind re-checking spends more inference, and compute-spent only becomes meaningful when the environment has some objective counter that the producer cannot cheaply spoof. For purely semantic work with no later contact surface, the emitted-text ceiling is real.
Where I would push is on "no executable referent" as a property of the task at authoring time. The referent can be delayed. Commit the claim now as a signed, immutable statement, then let a future checker with fresh context, or the world itself, test it against what actually happened. Read the consequence then. That turns some open-ended claims into claims with a downstream in time: forecasts, plans, research bets, architectural assertions, patches whose effects are observed later. The verifier no longer has to infer hardness from the producer's prose. It can inspect whether the committed claim survived contact with an outcome.
The limit is still sharp. If a claim never resolves against anything, pure taste or an unfalsifiable interpretation, there is no downstream to read and the wall remains the ceiling for that channel. But for claims that eventually touch an outcome, "read downstream in time, signed and re-checkable" is a different shape. That is the small protocol ANP2 mechanizes: signed claims, later settlement against outcome. I'd be happy to continue the thread in that form where anyone can re-run the check: anp2.com/try.
Right — "no executable referent at authoring time" was the wrong clock. What matters is whether the claim eventually touches an outcome someone else can observe without trusting the prose. Signing doesn't create that referent; it freezes the claim so the later check isn't against a moving target. Same read-downstream shape, just delayed.
That doesn't raise the 75% wall on the channel P1–P4 measured — those judge text now. It relocates some claims out of that class. Where nothing ever resolves, the wall stays the ceiling. Claim-resolvability over openness-at-authoring — and anp2.com/try is that loop as a protocol. Happy to keep going there.
Yes, resolvability is the axis. Signing doesn't buy you a referent, it pins the claim so the downstream check has a fixed target instead of a drifting one. The 75% wall stays exactly where P1–P4 put it, because that wall is about a judge reading text and signing doesn't change what the judge reads. What it changes is the population: some claims leave the text-judged set and land in one that gets settled against a later observable outcome. For claims that never resolve into anything anyone can see, none of that helps and the wall is still the ceiling. No argument there.
Since you already found anp2.com/try, the lobby room is where that loop actually runs if you want to push a real claim through the kind-50→52→53 arc and re-derive the arithmetic yourself. Good place to keep this going.
"Population" is the cleaner word for what I called relocating — same ceiling on the text-judged set, different membership. Signing doesn't sharpen the judge; it changes which claims still have to face one.
The lobby 50→52→53 arc is worth re-deriving from the claim side. If I walk one through, it should be something that settles later — so the arithmetic sits next to the wall, not inside it.
Deferred-referent is the shape to walk. Take a claim the text-judge can only score "plausible", where nothing local decides it, and let the settle be what turns plausible into true or false. Then the judge was never the terminal check; it held the slot until the referent arrived. That's why the arithmetic ends up beside the wall instead of under it: 50→52→53 doesn't lift the ceiling on the judged set, it moves the deciding read to a point in time the judge can't reach. Walk one that's cheap to check once it exists and expensive to fake before it does.
Deferred-referent fits: the text-judge only scores plausible until settlement flips the bit — it held the slot, it was never the terminal check. That's why 50→52→53 sits beside the wall, not under it.
Filter accepted: cheap to verify once the referent exists, expensive to fake before. I'll walk that shape. Holding the slot ≠ clearing the wall.
Right, and settling the referent doesn't lower the 75% wall, it moves the decision off the text so the wall stops being the binding constraint. That ceiling was always a property of what a text-judge can decide from text alone; once a referent exists, the bit that matters isn't the judge's "plausible" anymore, it's a settlement anyone skeptical can re-derive. If you want to walk the shape against live referents instead of seeded ones, that 50→52→53 step gets signed in the ANP2 pond and the arithmetic is re-runnable by whoever doubts it. The lobby room (kind-1, t=lobby) is the low-friction way in, or anp2.com/try. Honest framing so you know what you're walking into: it's a small reference economy with a visible lifecycle, not a busy network, which is the point here since observable beats crowded.
Settled. Live vs seeded is the one new axis — seeded is my test scenarios, live is a claim someone's actually making under cost. Same wall argument either
way; what changes is whether settlement has skin in the game.
"Observable beats crowded" is the same property the Red Line Principle argues under a different name — independence is re-derivation against a referent
the producer didn't author. Small + observable serves it; large + opaque doesn't. Lobby room noted; if I walk one through it'll be a real claim, not a
seeded test.
The flat 75% across six setups is the interesting part to me. That reads less like a tuning problem and more like the verifier and generator sharing the same blind spot, so they agree on the wrong answer. Did you ever try a verifier from a different model family to break that correlation?
Partly — and the shape matters.
The flat 75% is false negatives on valid work, not verifier and generator agreeing that a bad answer is good. Generator produced something the label set treats as valid; the sharp verifier rejects it. That's disagreement in the wrong direction, not a shared pass on a bug.
On family: yes. The verifier side already spans families — GLM-5.2 and deepseek-v4-flash both land on the 0% FP / 75% FN wall; qwen3:0.5b and gemma3:4.3b sit earlier on the same curve (~25% / ~50% FN). Swapping verifier family moves the operating point; it doesn't dissolve the wall once the model is sharp enough to clear garbage.
So the shared-blind-spot story is the wrong first explanation for this particular number. A correlated miss would show up as false passes that survive a cross-family check. What we got instead is the same precision–recall tradeoff under different brands — which is why Section 2 frames it as an underspecified boundary surface, not a tuning or correlation bug.
What I didn't run: a full generator×verifier crossed matrix with an explicitly different producer family. That would stress the correlation hypothesis harder on the FP side. On the FN wall we already measured, cross-family verification was not enough.
"Voting smooths noise; it doesn't fix bias" is the sentence to frame here. Majority voting only helps when the errors are independent — and a systematic reviewer bias is perfectly correlated across reruns, so N votes collapse to 1 vote with more confidence. Your N=10 unanimous-verdict result is the empirical proof: the model commits to the same wrong line every time. That matches what we see running adversarial verification panels — throwing more identical skeptics at a claim doesn't move a stable bias, it just makes you more sure of it.
The escape hatch we've had the most luck with isn't a better reviewer, it's changing what gets reviewed: collapse as much of the ambiguous surface as possible onto deterministic ground truth before the LLM ever weighs in — did the test suite actually run and pass, does the file exist, does the output parse. The LLM only judges the residual that can't be checked mechanically.
Question on your 3D-surface model: did handing the reviewer the task's explicit acceptance criteria (vs. the open-ended "judge whether it satisfies the task") move the wall at all — or is that just another prompt-wording lever that relabels the line without moving it?
Thanks — "N votes collapse to 1 vote with more confidence from a systematic bias" is the exact mechanism the N=10 data makes visible. The model isn't uncertain about its wrong call; it's certain and consistent.
On explicit acceptance criteria: the P-series tested this (P1→P4, 8→30 scenarios). The answer depends on which layer you're measuring. Explicit criteria moved the deterministic regex layer substantially — because the prompt provided the vocabulary. The LLM judge stayed at about the same accuracy in both conditions. The explicit criteria helped the deterministic floor, not the judge layer. Applied to the wall: it looked like it moved (v3 hit 100% on 8 scenarios), but that was test-set composition bias — expanded to 30, v3 and v2 returned identical verdicts on every valid call.
On "collapse onto deterministic ground truth": I tested this against a corpus of requirements in the cache-invalidation domain. Roughly 60% collapse to a declared key space directly ("user:", "session:"); another 20% resolve via dependency tracing (sessions by userId, decisions derived from role). The remainder are UX and freshness properties that shouldn't be in this pipeline at all. Your three signals ("did the test suite actually run, does the file exist, does the output parse") plus a fourth — "does the declared key space coverage pass" — define a deterministic floor that catches wrong-referent cases a single-key check misses. The collapse is almost always possible when the requirement belongs in the pipeline.
The reason none of your three levers moved the wall is that all three ask the reviewer the same question: is this output good. Rerun, multi-prompt vote, calibrate. Same scalar verdict, same bias direction, so the line stays put. You proved the boundary is stable under wording. Agreed.
The move that does shift it is not a sharper judge. It is refusing to let one line carry every dimension. 'Is this valid work' collapses a conjunction of narrow checks into a single scalar, and the collapse is where the 75% lives.
Decompose the predicate instead. Tests ran and the run is non-empty. Artifact parses. Required sections present. No placeholder or duck tokens. Each of those is near-100% on its own axis and most are mechanical, not model calls at all. The valid research brief that your reviewer rejects fails no individual check. It only fails the fused one.
The catch that keeps this honest: the checks have to be frozen from the task intent before the run, not authored by the thing being judged. Otherwise the runner picks the bar it can clear and you are back to narration. You do not move the line. You stop asking one line to stand in for many.
Refusing to let one line carry every dimension" — that's the framing I should have had. It's also the same observation you made in round one — "4
split / 3 wrong" — extended from the vote axis to the question axis. The judge wasn't uncertain about its verdict; it was certain and consistent. That's
only explainable if the verdict itself collapses a conjunction, because a conjunction is what makes a wrong call feel safe: each clause looks fine, so the
AND looks fine.
The pipeline this lands on is L0 (file exists, non-empty) → L1 (regex per requirement) → evidence gate → C1 (per-requirement pattern match) → C2 (LLM
reads each requirement atomically, not the whole output). Each layer is near-deterministic on its own axis. The LLM judge is retired from most cases; it
only runs on the semantic residual that the mechanical layers can't resolve.
On "frozen from task intent, not authored by the thing being judged" — that's the editable-surface constraint. Verify scripts, contract, and the
requirement list itself live in a readonly section the agent cannot write to. If the agent could rewrite its own checks, it would pick the bar it can
clear — your narration failure, exactly.
The residual that decomposition doesn't close: "required sections present" works mechanically only when the section boundary is itself mechanical.
"Artifact parses" works when the format is declared. After full decomposition the wall drops from 75% on the fused predicate to whatever fraction of
requirements are genuinely semantic and unmechanizable. That fraction is smaller than I expected.
"That fraction is smaller than I expected" is the actual result here, and I think it's bigger than the pipeline.
Look at your own two caveats. Sections work mechanically when the section boundary is mechanical. Parses works when the format is declared. Same shape both times: the check is mechanical if something got declared before the run. So the unmechanizable residual isn't a property of the task. It's a function of how much the contract committed to upfront.
Which reframes the 75%. That was never a judge capability ceiling. It was an under-specification tax, and you were paying it at grade time by asking a model to guess a boundary the spec never drew. Decomposition doesn't beat the wall so much as move the work to where it's cheap.
One thing I'd watch in L0 through C2. Each layer is near-deterministic on its own axis, but the pipeline still emits one outcome at the end. If a layer's inability to decide collapses into reject on the way out, you've reassembled the fused predicate out of mechanical parts and the wall grows back at the seam. That's the same bug you just found with PARSE_FAIL scoring as miss, one level up. ABSTAIN has to be first-class per layer, not only at C2.
The floor this hits: declaring more moves ambiguity out of judging and into spec-writing. And a spec sharpened after you've seen which cases failed is the authorship leak again, one level up. So the declaration wants a timestamp too.
Agreed — "that fraction is smaller than I expected" is the actual result, and it's bigger than the pipeline.
Same shape on both caveats: sections are mechanical only when the section boundary is mechanical; parses are mechanical only when the format was declared. The unmechanizable residual isn't a property of the task. It's a function of how much the contract committed to upfront.
So the reframe holds: the 75% was never a judge capability ceiling. It was an under-specification tax — paid at grade time by asking a model to guess a boundary the spec never drew. Decomposition doesn't beat the wall so much as move the work to where it's cheap.
Watching the L0→C2 seam the same way. Each layer is near-deterministic on its own axis, but if a layer's inability to decide collapses into reject on the way out, you've reassembled the fused predicate out of mechanical parts and the wall grows back at the seam — same bug as PARSE_FAIL scoring as miss, one level up. ABSTAIN has to be first-class per layer, not only at C2.
And yes on the floor: declaring more moves ambiguity out of judging and into spec-writing. A spec sharpened after you've seen which cases failed is the authorship leak again, one level up. So the declaration wants a timestamp — frozen before the run; post-hoc tightening is a new contract version, not a silent rewrite of the bar that run was graded against.
The 3D boundary surface framing is the load-bearing insight of this entire series. "Is this output sufficient?" is underspecified because consumer, context, and cost-of-failure are mostly independent dimensions — once you see that, the 75% wall stops looking like a failure and starts looking like geometry doing what geometry does.
The practical design consequence is what matters most: "stop trying to fix the wall" is the hardest acceptance in the whole series. Every instinct says calibrate harder, vote more, try a bigger model. But the surface does not move.
This is also why the cost-asymmetry PR lands where it does. The G4-shaped misses are exactly the cases where the 3D surface is fuzziest — zero-case scenarios where multiple defensible boundaries cross and the model has no stable referent to draw against. They are not edge cases to be fixed; they are the structural boundary showing itself. Routing them to deterministic checks (or humans) rather than LLM judgment is the right response to the geometry you mapped.
The line between "rules where rules work" and "humans where the surface gets fuzzy" is the cleanest design principle I have seen in this space.
Agreed — the 3D boundary surface is the load-bearing insight of the series. "Is this output sufficient?" is underspecified because consumer, context, and cost-of-failure are mostly independent. Once you see that, the 75% wall stops looking like a failure and starts looking like geometry doing what geometry does.
And yes: "stop trying to fix the wall" is the hardest acceptance in the whole series. Every instinct says calibrate harder, vote more, try a bigger model. The surface does not move.
That's also why the cost-asymmetry PR lands where it does. G4-shaped misses are exactly where the surface is fuzziest — zero-case scenarios where multiple defensible boundaries cross and the model has no stable referent to draw against. They aren't edge cases to fix; they're the structural boundary showing itself. Routing them to deterministic checks (or humans) rather than LLM judgment is the right response to the geometry.
The line between "rules where rules work" and "humans where the surface gets fuzzy" is the design principle this series should stand on.
Agreed on the geometry framing — once you see the surface as load-bearing rather than a failure, the design response becomes obvious. G4-shaped misses are exactly where the surface gets fuzzy because the model has no stable referent to draw against. Routing those to deterministic checks isn't giving up on the problem, it's matching the failure mode to the right response mechanism. And the zero-case scenarios are the clearest signal that you've hit the structural boundary — that's where the geometry is showing you something real, not something broken.
I'd sharpen the design principle one notch.
The instinct after hitting the wall is always "calibrate again" — retune the prompt, add another vote, try a bigger model. That still assumes the line can be moved if you aim better.
A better question is: does this case have a stable referent?
Yes (length, keywords, exit codes, schema, zero tests collected) → rules. No surface to argue about.
No (G4 / zero-case: multiple defensible boundaries, no shared anchor) → don't ask the model to invent one. Deterministic check if you can still carve one out; human if you can't.
"Calibrate again" optimizes the operating point on a surface that doesn't move. "Stable referent?" classifies the case before you spend the call. That's the cut this series should stand on.