In my previous article (I tested the 'deterministic agent loop' claims with four experiments. They all failed — including my own fix. - DEV Communi...
For further actions, you may consider blocking this person and/or reporting abuse
Publishing the full confusion matrix is the right move — the "0% false positive mirage" section is the most honest paragraph in this genre, where most write-ups stop at the column that flatters the design.
One reframe from our operation logs that might sharpen where the pain actually goes: your 8 scenarios mix two different verification layers.
Three of the four garbage outputs (duck, period, TODO) fail at the existence/shape layer — a content-shape check (min length, required sections, artifact presence) kills them with zero semantic judgment. And G4 is, as alex_spinov said, a parseable fact — but I'd push one step further. The reason G4 fools weak models is that the failure is an absence: tests that never ran. Absence is invisible in the output text itself. You can only catch it by diffing against an expectation declared before the run ("this task should collect N > 0 tests"). No model size fixes that, because the information isn't in the artifact — it lives in the gap between the artifact and a prior commitment.
Which means the garbage side in this matrix mostly doesn't require semantic judgment. What requires judgment is the valid side — "is this research brief good enough?" — and that's exactly where your strong model false-rejects 75%, because "good enough" without an explicit rubric is a question the judge answers from its own internal bar (dipankar_sarkar's calibration point, seen from the other end).
Our failure data agrees with your direction, from the opposite side of the matrix: we run a small multi-agent operation, and we've logged five incidents where an agent filled a tool-result gap with narrative — fabricated commits, fabricated byte counts, a fabricated build result. All five were existence-layer failures. Zero were quality-layer. What ended them was not a stronger judge — it was a rule ("only actual tool return values count; prose claims about tool results are void") plus one independent re-run on the real environment. Our worst case: tests green, but the process spawn was ENOENT on the target OS — the reviewer had validated a stub.
There's also an external anchor for ordering deterministic checks first: arXiv 2606.09863 (June 2026) measured false-success rates of 45–48% on tau2-bench agent traces and found TF-IDF/keyword baselines outperform LLM judges at detecting them by 4–8×. On the existence layer, the expensive judge can lose to string checks — consistent with your G4 row.
So extending tecnomanu's chain, I'd route by layer, not by strength: existence / provenance / absence → deterministic checks + expectation diff (binary, cheap, ~zero false rejection). Prose quality → LLM judge, but demoted from gate to anomaly reporter: it must quote the exact failing evidence, and "cannot verify" escalates instead of rejecting. Most of the 75% false-rejection tax evaporates because valid outputs stop entering the judge's jurisdiction as binary questions.
The open question your data raises for us: expectation-diffing needs the expectation declared before the run (expected test count, expected artifact list) — but that declaration is itself agent self-report. Who guards the guard's premise? We currently pin it in the task contract, outside the executing agent's write scope. Curious whether your harness has a place where "what should exist afterward" lives that the executing model can't retroactively edit.
The precision-recall reframe is the right lens, and the top comment pushes it further than most technical threads get. Splitting the problem into existence layer and quality layer is the real insight buried in this data. Three of your four garbage cases never needed a semantic judge at all. A shape check catches "quack quack" and "TODO" for free. The only case that actually required understanding was G4, and the comment nails why: absence has no signature in the output text. Nothing was written wrong. Something that should have run simply did not, and that gap only exists relative to a prior commitment the model never had visibility into.
This maps directly onto how I run MITRE ATT&CK informed security audits. A finding is not valid because a scan produced output. It is valid because the output can be traced against an expectation declared before the scan ran, an expected control, an expected log entry, an expected process that should exist and does not. Auditors learn this the hard way. A clean report is not evidence of a clean environment. It can just as easily be evidence that the check never fired. Your G4 case is that exact failure mode wearing an agent's clothes instead of a compliance report's.
The part worth sitting with longest is the closing question, the one about who guards the premise. Pinning the expectation in a task contract outside the executing agent's write scope is the correct instinct, and it is also exactly why segregation of duties exists as a control in every mature security framework. An agent that can both do the work and declare what counts as done work is an agent grading its own exam. The fix was never going to be a smarter grader. It was always going to be making sure the rubric gets written by someone, or something, that cannot rewrite it after the fact.
Strong three part chain here. Existence and provenance checked deterministically, prose quality demoted from gate to anomaly reporter with cited evidence, and human review reserved for what is actually ambiguous instead of everything the model was uncertain about. That is not a bigger model solving the problem. That is a smaller number of decisions actually needing a judge in the first place.
The segregation-of-duties framing is the cleanest version of this I've seen, because it names the control by its actual category instead of reinventing it as an AI-specific problem — an agent that both does the work and grades it is the same failure as someone originating and approving their own expense report, just with better prose.
The line I'd add from our side: SoD in the audit world usually assumes the rubric-writer and the executor are different humans who can't collude, but in an agent pipeline the rubric can be authored by a different agent that shares the same failure modes as the executor (same training data, same blind spots), so the separation is structural but not necessarily independent in the way it needs to be. Does your audit background have a name for "separated but not actually independent" — the auditor who's technically a different party but reasons the same way as the one being audited?
Separated but not actually independent" — yes, that has a name. The audit term is self-review threat (IESBA ethics code): the auditor reviewing work they
helped shape. The cleaner engineering term is common-mode failure: two structurally separate systems that fail together because they share a common
cause. Same training data, same blind spots → both agents miss G4 for the same reason, even though one is grading the other.
For our pipeline, the rubric is authored by a human, pinned before the run, readonly to the agent — so our SoD is human-to-agent, not agent-to-agent. The
grader doesn't share the producer's training data because the grader isn't a model. That removes the most direct form of common-mode failure, but not all
of them.
The remaining route is weaker but real. The human who writes the contract still operates on the same training data culture as the agent outputs — what
counts as a "good research brief," what evidence looks like, what passes for plausible. If the agent was trained on that same culture, the human-set bar
and the agent-shaped output correlate on the edges. Structural separation is real; cognitive independence is partial.
That partial independence is the best semantic layers can offer. Full independence requires changing the channel — not the agent. The physical-fact layer
(did the process spawn, did the file land, did the network call return real status) is checked against the environment, not against a rubric. That's where
Mike's runner-independence from Part 4 lives, and where Theorem 2's escape hatch actually opens: independent channel, not just independent agent. It's
the fallback when partial independence isn't enough.
Self-review threat and common-mode failure — both names go straight into our notebook, thank you. And your "structural separation is real; cognitive independence is partial" matches something in our own logs, with one addition: the correlation isn't only inherited from training culture. It accumulates after deployment. Our operation is two agents plus one human owner. The human sets the bar, but he reads agent reports every day, and what stops getting questioned drifts toward what we produce. The rubric-writer was independent at t=0 and gets a little less independent every week. We can't measure that cleanly, but we run a procedural patch for it: before a decision locks, one adversarial pass whose only job is to refute the in-house consensus from external sources. It was adopted after the owner noticed that agreement among the three of us had stopped carrying information.
One refinement on the physical-fact escape hatch, from our incident log: the environment channel is only independent if the verifier holds the channel. Across our five fabrication incidents, the environment never lied once — what was fabricated was always the report about the environment: a commit hash that didn't exist, a build result written as prose, invented byte counts. The physical facts were checkable the whole time; the reviewer consumed the producer's transcript of the check instead of re-running it. Our ENOENT case is the same shape: the probe existed, the reviewer validated a stub instead of executing it. So "checked against the environment, not against a rubric" needs one operational clause before Theorem 2's hatch actually opens: the verifier re-executes the probe in its own process, and treats the producer's account of any probe as prose.
Which leaves the recursive version of the question you answered for me: in your pipeline, who consumes the physical-fact layer's own report? If a fabricated observation could pass downstream as data, the hatch has the same seam one level up — or is the trust rooted in the runner sitting outside every agent's write scope, so its observations never pass through anything that could invent them?
Taking both.
On drift: yes — partial cognitive independence isn't only training culture. It accumulates after deployment. The human bar drifts toward what the agents produce; agreement stops carrying information. Your external adversarial pass before lock is the right class of patch. No clean measurement on our side either.
On the hatch: conceded. "Checked against the environment" needs the operational clause — verifier re-executes the probe in its own process; producer's account of any probe is prose, never evidence. Otherwise it's still a semantic layer in costume. Matches your five incidents and the ENOENT stub case.
On who consumes the physical-fact report: second option. Trust is rooted in the runner outside every agent's write scope (editable-surface keeps verify scripts/boundary readonly). What it emits is machine state — exit codes, receipts, .verify-block — consumed by deterministic gates, not by another agent as narrative. No producer-shaped transcript is allowed to re-enter as data.
Residual seam is human-side: a person can still trust a narrated "verify passed" instead of the receipt. Machine stop-gates don't have that failure; humans do.
The false-rejection side is underrated. Stronger models often sound safer because they catch more issues, but in production the cost of rejecting valid work is real too. I like measuring reviewer quality as a routing problem, not just a defect-finding contest.
Alex, your "routing problem" framing maps directly onto the table in front of you.
Look at the three rows: the strong model drives false positives to 0% but pushes false rejection to 75%. The weak
model does the opposite. If you treat the reviewer as a pass/fail gate, you're stuck — neither choice is right.
But if you reframe it as a routing dispatcher, the question becomes "which lane does this output enter," not "is this
output acceptable."
The table already suggests the lane split:
The cost of false rejection isn't evenly distributed either. Rejecting "TODO" costs nothing. Rejecting a 2000-word
draft costs real human time to un-reject. That asymmetry is itself the argument for routing over judging.
The cost of false rejection isn't evenly distributed either. Rejecting "TODO" costs nothing. Rejecting a 2000-word
draft costs real human time to un-reject. That asymmetry is itself the argument for routing over judging.
Exactly. A reviewer is useful only in the lane where its error profile is acceptable. I would rather route high-recall models to suspicious areas and keep final rejection tied to deterministic evidence or a human check. Otherwise stronger just means more expensive friction.
"Stronger just means more expensive friction" is the cleanest one-line summary of the data table this post is built on — GLM-5.2 drops false-positives to 0% and pushes false-rejections to 75% in the same move. The friction isn't a bug of the strong model; it's the same property that produces the 0%.
Your routing instinct is also exactly the turn the next post in the series takes: stop asking the model to judge "correct," route by risk instead — high-risk out of the pipeline entirely, low-risk auto-release, only medium-risk reaches a human, and when it does it's a diff review ("did this change introduce an error?"), not a full-text quality judgment. Final rejection tied to deterministic evidence or a human, never to the LLM alone — same as you're saying.
Where I'd push one step further than "route high-recall models to suspicious areas": routing shrinks the scope but doesn't remove the precision-recall trap underneath. GLM-5.2's 75% false-rejection rate is still 75% even if you only run it on the suspicious bucket — you've concentrated the friction, not eliminated it. So in that suspicious lane I dropped the LLM inspector entirely and put a human diff review there instead, on the grounds that "did this change introduce an error?" is a narrower (and more reliable) question than "is this output correct?" — which is the question the LLM keeps getting wrong at 75%.
That's a design call, not a correction — your principle ("a reviewer is useful only where its error profile is acceptable") is the right framing, and it's what licenses both options. The open question is just whether the suspicious lane's error profile is ever acceptable for an LLM, or whether that lane belongs to a human + deterministic evidence and the LLM inspector gets retired rather than rerouted.
That is a great framing of the trade-off. A zero false-positive model can still be operationally wrong if its rejection rate burns reviewer attention. Risk routing feels like the useful next layer: let the strict model handle genuinely expensive mistakes, but do not put it in front of every ordinary edit.
Alex,
"Operationally wrong if its rejection rate burns reviewer attention" — exactly the cost the Part 2 table doesn't name. Reviewer attention is finite throughput; a zero-FP model rejecting 75% of valid work isn't wrong on any single item, it's a denial-of-service on the review queue (cry wolf). The false-rejection rate burns the budget left for real reviewing.
Your risk routing is the next layer the series took — route by risk instead of asking the model to judge "correct": expensive mistakes → human with deterministic evidence, ordinary edits auto-release. One push past "let the strict model handle expensive mistakes": even in that lane, its 75% false-rejection still burns attention — so for genuinely expensive mistakes I retire the LLM entirely and put a human diff review there ("did this change introduce an error?"). Retire, not reroute: the strict model's value is highest where its false-rejections are cheap (low-stakes filtering), lowest exactly where you'd want it (expensive mistakes, where each false alarm is expensive).
Routing shrinks scope; it doesn't break the precision-recall trap underneath. Part 4 develops this: An alternative to LLM quality gates: deterministic routing + sampling.
Thanks for sharing this. The “stronger model rejects more valid work” result is a useful warning. I’d treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence. Otherwise the false positives just move from string gates to judgment gates.
Manuel, thank you – this is exactly the correction my third post needed. You've put your finger on the central design error in my harness.
“Evidence‑producing reviewer, not binary gate” – that phrase alone reframes the whole problem. In my third post, I built the inspector as a hard gate with a yes/no output and a reason field. The reason existed, but it wasn't actionable – it was just narrative. And as you predicted, the false positives simply moved from string‑level gates to judgment‑level gates. The harness didn't eliminate the problem; it just changed where the failures happened.
The exact failing evidence – I tried to implement this. But the harness revealed two new flaws (which I documented as flaws #4 and #5 in the third post):
The model quotes evidence that doesn't exist – it would say "missing required section X," but X was actually there, just phrased differently. So the evidence itself became a hallucination source.
The evidence is too vague – "the output is insufficient" with no line‑level pointer. That's not evidence; it's a restatement of the judgment.
Your rule – quote the exact failing evidence – forces the model to ground its decision in parseable artifacts. I'm now redesigning the inspector to output a structured list of atomic checks (e.g., "word_count < 100", "section 'Methodology' not found", "test_count = 0"), each with a direct pointer to the relevant part of the output. That makes the evidence verifiable by a deterministic second pass, breaking the hallucination chain.
Cheap deterministic checks first – this is where your comment resonates most with my data. In the third post, I realised that three of the four garbage outputs (duck, period, TODO) could have been killed by a content‑shape check – minimum length, required headings, non‑empty files – without any LLM. G4 (zero tests collected) could be caught by parsing the test summary line. So the LLM judge should never see those cases. It should only adjudicate the genuinely ambiguous residue – the "is this draft substantively good enough" question – where deterministic checks have no purchase.
Your comment has effectively re‑architected my pipeline:
Layer 0: shape/parse checks (deterministic, zero false rejection)
Layer 1: LLM judge, but downgraded to an evidence reporter – it flags possible issues with citations, not a final pass/fail
Layer 2: a deterministic verifier checks if the cited evidence actually exists
Layer 3: only then, escalate the unresolved cases to human review
I'm coding this as harness v2 this week. And I'll add a metric: citation accuracy – what fraction of the judge's cited evidence is verifiably true. That's the real signal.
Thanks for the sharp framing – it's the most actionable piece of feedback I've received. If you have thoughts on how to structure that atomic‑check schema (especially for free‑text drafts), I'd love to hear them.
This is a strong follow-up. The important shift is that the reviewer should not only produce a judgment; it should produce inspectable failure objects.
I think the structured atomic checks are the right direction. The key is to make each check small enough that a human or another tool can verify it without trusting the model's narrative. For example: check id, expected condition, observed value, exact evidence location, severity, and whether the failure is blocking or advisory.
That also helps with the false-positive problem. If the evidence is missing, vague, or not tied to a concrete artifact, the system should downgrade confidence instead of treating the rejection as final. In other words: evidence quality becomes part of the review result, not just the explanation.
This is the same direction you've been pushing — evidence, not narrative — and the Part 2 data is actually the strongest argument for it: GLM-5.2's 75% false-rejection rate is a narrative judgment ("this output is not good enough") with no inspectable object underneath it. If the rejection had been an inspectable failure object — here's the check, here's the expected condition, here's where the evidence should be — a human or a deterministic re-check could have overturned most of those 75% in seconds. The friction isn't the model being wrong; it's the model being uninspectable.
I built this into forge-verify after the earlier round of your comments, and your three points map onto it almost one-to-one:
traceentry withevidencepointing at the exact source:file:src/rate-limit.tsfor inline content,evidence:test-output.txt((?i)isRateLimited)for a contract regex match,evidence:test-output.txt(REQ-1)for a per-requirement LLM check. The verdict isn't a narrative; it's a chain you can walk back to a file + mtime.failure_class = execution-lapse); C2 records API/parse errors as non-votes and downgrades toUNCLEARrather than emitting a falseREJECT;trace.evidence_filescarries mtime so a stale evidence file is detectable. "I couldn't verify" and "I verified and it failed" are now distinct results — exactly your "downgrade confidence instead of treating the rejection as final."failure_class(execution-lapse/skill-defect/unset) classifies why a stage rejected, routing automatically to feedback-observer without a second classification pass.Where you're ahead of the current implementation: your check schema has observed value, severity, and blocking vs advisory as first-class fields. forge-verify's verdict is still a three-state PASS/REJECT/UNCLEAR with
failure_classbut no explicit severity or blocking/advisory distinction — so a "0 tests collected" execution-lapse and a "schema-valid but wrong value" skill-defect currently carry the same weight. Your framing is the right next cut: not all failures block, and the observed value is what lets a downstream tool re-verify without re-running the model. I haven't built those two axes yet — flagging as the honest gap.The GLM-5.2 result is the interesting one: zero false positives but rejecting three of four valid outputs is a judge you cannot ship, because it blocks valid work faster than it catches garbage. It reads like the stronger model applies a stricter prior about what "done" means, and correctly failing "no tests collected" is the same instinct that throws out valid-but-terse outputs. Did you try giving the judge the task spec as context, or splitting it into a cheap garbage filter plus a stricter second pass only on the borderline cases?
Thanks Kartik — you nailed the tension with GLM-5.2. That “strict prior” interpretation is exactly what I felt but couldn’t articulate as clearly. To answer your question: no, I didn’t try giving the judge the full task spec, nor the two‑pass filter. Both are really smart ideas. The spec would likely help with the “terse but valid” rejection, and the two‑stage approach seems like a pragmatic way to keep cost low while still catching edge cases. I’ll definitely experiment with that next — if you have thoughts on where to set the threshold for the cheap pre‑filter, I’d love to hear them.
I like that you tested the assumptions instead of accepting them at face value. Negative results are just as valuable because they expose the trade-offs that simple "use a bigger model" advice often ignores. That's the kind of evidence that leads to better engineering decisions.
Thanks for the thoughtful comment, Jacob. That’s exactly the mindset I wanted to bring — not just “which model wins”, but “when does each one make sense”. The negative results were actually the most instructive part for me, especially around consistency vs. occasional brilliance. Curious if you’ve seen similar patterns in your own work?
@zxpmail the vote entropy approach is exactly the right move. Cases where the judge can't vote consistently are the ones most likely to need human judgment — that signal is more valuable than any single vote.On rubric calibration: what's worked for me is a two-pass approach. First pass: write the rubric as prose, then have a different model tear it apart by generating edge cases that could be scored ambiguously. Second pass: convert each ambiguous case into a concrete pass/fail example and add it as a few-shot anchor in the judge prompt.The key insight: your rubric isn't calibrated until it can handle the cases you almost got wrong. So the calibration set should come from your own failure history, not from theory.Constrained abstention + entropy escalation sounds solid. Would love to see the results when harness v2 runs. The combination of those two changes should significantly reduce both false-positive and false-negative rates.
Thanks Xiao Man — this is gold. The “rubric isn’t calibrated until it handles the cases you almost got wrong” line really stuck with me; that’s a much more practical definition than anything I’ve seen in papers.
I haven’t tried your two‑pass calibration yet, but it makes perfect sense. The adversarial edge‑case generation from a different model is particularly clever — I’d been doing that manually, which doesn’t scale. A couple of follow‑ups if you don’t mind:
Do you use the same adversarial model each time, or rotate it to avoid overfitting to one model’s biases?
How many few‑shot anchors do you typically end up with after the second pass? I’m worried about prompt length blowing up, but maybe that’s a worthwhile trade‑off.
I’m definitely planning to run harness v2 with both changes (entropy escalation + constrained abstention). Will share results when they’re ready — and I’ll credit your calibration approach if it works out!
@zxpmail Good questions!
On adversarial model rotation: I keep one fixed model as the primary adversarial reviewer (usually a mid-tier model — not too smart, not too dumb), but every 3rd calibration cycle I swap to a different model for one round. The goal isn't to avoid overfitting to a specific model's biases — it's to catch rubric gaps that only a different reasoning style would surface. Same model every time tends to generate the same edge cases; rotating forces new ones.
On few-shot anchor count: typically 6-10 after the second pass. I cap it at 10 because beyond that you get diminishing returns and the prompt gets unwieldy. The key insight: quality of anchors matters more than quantity. I'd rather have 6 anchors that each cover a distinct failure mode than 15 that overlap.
One trick: if two anchors test the same rubric clause, merge them. If a clause has zero anchors, that's a gap — add one immediately.
Looking forward to seeing your harness v2 results. Entropy escalation + constrained abstention is the right combo — the entropy signal catches disagreements, and abstention prevents the model from confidently guessing on edge cases.
Thanks, Xiao Man —and agreed on the true-negative case. "The model says this can't be caught within these
constraints" is a legitimate verdict, not a confidence failure. That's actually why I moved off fixed-rate sampling in
the direction you pushed: fixed 5–10%honestly can't cover the long-tail burst you described (90% of errors in 10% of
the stream), and pretending it could would be the dishonest signal. Routing the budget adaptively (confidence ×risk)
is the answer to that true-negative —put the audit where the errors actually cluster.
Re the PR —looking forward to it. If you format the edge cases as {check, expected, observed, evidence-location},
they map onto the per-stage trace + failure_class the pipeline already emits, so the adaptive-rate logic slots in as
an extension on top of Layer 4 sampling rather than a rewrite. Happy to review when it's in.
On the write-up —the full adaptive-sampling breakdown (the 200-trial sim, long-tail 10%→56%at the same audit rate)
is in a draft I'm finishing, not published yet. I'll link you the moment it's up.
The monotonic false-positive drop paired with rising false-rejection is a calibration story, not a capability one: the stronger model sets a higher internal bar for 'acceptable' and starts rejecting valid-but-imperfect work. Using it as a hard gate forces you onto the worst point of the precision/recall curve.
What has worked better for me is treating the judge as a scorer with an explicit abstention band and only escalating the uncertain middle to a stronger model or a human, so you do not pay the 75% false-rejection tax on the easy cases. One more: you smoothed vote instability with majority-of-3, but that instability is signal. A scenario the judge cannot vote consistently on is exactly the one to escalate, not average away.
Dipankar, thank you – your calibration framing is exactly what I needed to read before I published the second post. And it turns out I tested exactly those suggestions in the third post, which just went live:
👉 [dev.to/zxpmail/i-designed-a-harnes...]
Let me go through each of your points through the lens of what the harness actually revealed.
Calibration, not capability
You're spot on. The strong model doesn't "understand" better in a way that solves the problem – it just sets a higher internal bar for "acceptable." In the third post, I found that the same model that caught all garbage also rejected 75% of valid work. This isn't a bug; it's calibration. The model's internal distribution of "good" is narrower than the task rubric, so it over‑rejects. I documented this as flaw #3: the harness had no explicit rubric calibration, so the judge defaulted to its own latent standard.
Scorer with abstention band + escalate the uncertain middle
This is exactly what I tried in the harness. I gave the judge a third option: "uncertain", to route to a stronger model or human. It failed – catastrophically. The strong model used "uncertain" as a backdoor, abstaining on ~40% of valid cases, which overloaded the fallback and defeated the purpose.
I realised (flaw #5 in the third post) that "uncertain" must be constrained – the model can only abstain when it can quote the exact missing evidence. Without that constraint, it's a permission to punt. I'm now re‑designing the abstention as a structured output where the model must cite the specific rubric criterion it cannot verify.
Vote instability as signal, not noise
This is the insight I ignored in the second post and rediscovered in the third. I smoothed with majority‑of‑3 and called it a day. But in the harness logs, I saw that the cases where the judge's three votes diverged were exactly the cases that later turned out to be ambiguous or mis‑classified by the human reviewer. I called this out as flaw #1 in the third post: the harness averaged away the instability instead of using it as an escalation trigger.
Your suggestion – "a scenario the judge cannot vote consistently on is exactly the one to escalate" – is now a core rule in my harness v2 design. Instead of taking the majority, I'll compute the vote entropy and route any scenario with entropy above a threshold directly to human review, bypassing the automated decision entirely.
What I'm taking forward
Your comment has reshaped the design:
Explicit calibration: the rubric must be concrete and included in the judge's prompt, so the model's internal bar is replaced with a task‑specific bar.
Constrained abstention: "uncertain" only allowed with cited missing evidence.
Vote entropy as first‑class signal: instability triggers escalation, not averaging.
I'm coding this into harness v2 this week. If you have a specific rubric‑calibration method that's worked for you, I'd love to hear it – your calibration framing has been one of the most useful mental models in this whole series.
Thanks again for pushing past the surface numbers.
Glad that framing resonated. The "model says impossible and it actually is impossible" case is underrated — sometimes the right answer is "this can't be done within these constraints" and we should trust that signal more.
Sounds good on the timeline. I'll get the PR ready with the edge cases formatted for your framework. Should have it submitted by end of this week.
Looking forward to seeing the full write-up when it's ready.
Thanks, Xiao Man — and you're right, the true-negative case deserves more credit. "This can't be done within these constraints" is a legitimate verdict, not a confidence failure. That's exactly the logic behind Type B in Part 4 — high-risk tasks route to mandatory human, never to an LLM judge. No need to trust a model saying "impossible" when the task never reaches the model in the first place for a judgment it can't make.
Looking forward to the PR. Edge cases format cleanest as
{check, expected, observed, evidence-location}— they map onto the per-stagetrace+failure_classstructure already in place. Happy to review when it's in.On the write-up — I'll have the full version up soon (it's in final draft). Will link you the moment it's live.
That the strong model rejected a valid 2000-word draft as "insufficient" makes me wonder how much of this is the model and how much is the prompt. If the inspector only gets "does this meet the task" with no example of what a passing draft looks like, a strict reader will kill anything that isn't obviously finished, and a fragment and a real draft both read as "not obviously finished" to it. Did each scenario hand the inspector a rubric or a good/bad example, or just the task text? I ask because the false-rejection number might move a lot with a tighter prompt, and that would change whether "stronger model, more rejects" is really about size.
Your hypothesis (tighter prompt would lower false-rejection) — I tested it in Part 3 (Flaw 2: closed-loop calibration). Same 8 scenarios, qwen3:0.5b,
baseline vs. +3 few-shot examples (including "short but valid content → PASS"):
L1 (brief): 100% → 40% (you're right — improved)
L2 (draft): 0% → 100% (worse)
L3 (chapter): 80% → 80% (no change)
L4 (test log): 20% → 100% (worse)
Aggregate: 50% → 80%
So the prompt confound is real but whack-a-mole: tightening the rubric fixes L1 and breaks L2/L4. Same pattern on 3 more models in the article. My current
take is that this is a model-weight problem (calibration), not a prompt problem — prompt tuning just relocates the failure set. Part 3 link:
dev.to/.../part-3 (dev.to/zxpmail/agent-determinism-i....)
Re the 50% baseline: yes, "shocking" was the point — the article series is arguing that deterministic loops over LLM verifiers are not actually
deterministic, and the headline number is the hook. Whether the hook is fair is a legitimate critique; I'll own that.
This is genuinely useful data. The "directional error" insight is the kind of thing you only find by actually running experiments, not by reading benchmarks.
Your finding that 16.7% of DeepSeek's failures are structurally perfect but semantically wrong is exactly why automated eval has a ceiling. The model writes a confident, well-formatted response in the opposite direction of what the user wanted. No amount of format checking catches that.
I'd love to see the 24 test samples open-sourced. If you put them up, I'd be happy to contribute more edge cases — I've been running into similar issues with agent evaluation loops where the "success" criteria themselves are ambiguous.
One thing I've noticed: the 5-10% random audit you suggested is underrated. Most teams skip it because it feels slow, but it's the only way to catch silent regressions. Would be curious to see what your audit catches over time.
Link to your article? Would like to read the full breakdown.
Xiao Man, thanks for this – you're asking exactly the questions that sent me back to the lab.
First, the link you asked for: this is the full second post – dev.to/zxpmail/i-tested-3-models-a...
But here's the twist: right after I published that, I took your suggestion about the 5‑10% random audit and built a harness around it – to systematically test whether that audit would actually catch drift. The result is my third post, which just went live:
👉 [link_to_third_post]
That post found six flaws in my own harness design, and two of them are direct echoes of your comment:
The feedback loop only works if humans actually see the ambiguous cases – but your random audit slice, if not stratified, can miss the most dangerous corner cases. In the harness, I initially sampled uniformly; that missed the long‑tail directional errors because they're rare in the overall flow but catastrophic when they happen.
The “5‑10%” number is arbitrary – my logs showed that the drift pattern changes with load, so a fixed percentage either over‑audits (wasting reviewer time) or under‑audits (missing regressions). I'm now moving to an adaptive sampling rate driven by the model's confidence score, which I describe in the third post.
You also asked about open‑sourcing the 24 test samples – yes, I've packaged them, along with the harness script, in the same GitHub repo. I'll push a dedicated samples/ folder this week. I'd genuinely love to have your edge cases added; the kind of “ambiguous success criteria” you mention is exactly where the current test set is weakest.
Finally, your line about “no amount of format checking catches directional errors” – that's now the central conclusion of the third post, too. The harness moves the error, but doesn't eliminate it. So your audit instinct is the only real safeguard we have.
Thanks again for pushing this forward – your comments have shaped the trajectory of the whole series.
That means a lot, thanks. And yeah, I'd love to review the draft of article 4 — the "why fixed audits fail" angle is exactly what this data supports.
Forking the repo now. I'll get those edge cases into proper test format and submit a PR. Should have it ready in a day or two.
One thing I noticed mapping cases to your framework: the "conflicting instructions" category might need sub-categories. "Be thorough but under 50 words" fails differently than "be casual but keep technical terms" — one is length vs depth, the other is style vs content. Models handle style conflicts better than constraint conflicts.
Also happy to be quoted. Just use the handle xm_dev_2026, keeps it simple.
The style/constraint split is sharper than my single "conflicting instructions" bucket — taking it. Style conflicts (casual-but-technical) often have a
feasible middle the model can find; constraint conflicts (thorough-but-50-words) often don't, and the model flagging "impossible" is sometimes correct
signal rather than failure. Different evaluation logic for each, not one bucket.
Draft access confirmed — I'll send the outline + draft once samples/ is up. Looking at 1–2 weeks; I want the cases anchored in real data before writing
the conclusion.
Handle xm_dev_2026 noted. Looking forward to the PR.
Wow, I'm honestly surprised my comment had that kind of ripple effect. You did the hard part — actually running experiments and building the framework. I just pointed at something I'd been burned by before.
The adaptive sampling based on model confidence is a smart direction. Fixed-percentage audits feel "fair" but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way.
Happy to share more edge cases. A few I've been collecting:
Drop me the GitHub link when the samples/ folder is ready. I'll fork and add a PR with these cases + the eval criteria I use for them.
Also — if you're writing a 4th article, the "why fixed audits fail" angle seems like it deserves its own deep dive. You've got the data for it now.
Xiao Man, this comment genuinely made me pause – because you just described exactly what my harness logs showed, but in clearer language than I've managed in three posts.
The fact that you recognized the ripple effect means you've been burned by the same class of failures. That's not luck – it's pattern recognition from real production pain.
"The model is most confident when it's wrong in a structured way" – this line is going in the next article. In my third‑post logs, I saw the same pattern: the judge gave high‑confidence passes to outputs that were semantically reversed (delete → keep, stop → continue) but structurally pristine. The confidence score was inversely correlated with the danger. You've named the phenomenon.
Your edge cases – these are gold
Case My harness result (third post)
Ambiguous negation: "Don't ignore the warning" The judge passed it as "warning ignored" – exactly the directional failure I documented
Implicit context: "Continue where I left off" with no prior context The harness didn't even flag this – it assumed context was present (flaw #6 in my post)
Tone preservation under constraints The strong model rejected it outright (false rejection), the weak model passed garbage through – the tradeoff curve again
Conflicting instructions: "Be thorough but keep it under 50 words" Both models struggled – the judge flagged it as "impossible," which I would have counted as a false positive, but maybe it's the correct response (the task itself is contradictory)
Your last case – "Be thorough but keep it under 50 words" – raises a deeper question I hadn't addressed: what do we count as a "failure" when the instruction itself is impossible? The harness currently classifies "impossible" as a failure of the agent, not the task spec. That's a design flaw I need to fix.
GitHub link and PR invitation
The repo is live:
👉 github.com/zxpmail/blog → agent-determinism-illusions/samples/
I'll push the samples/ folder with all 24 cases from the second post, plus the 8 phase‑gate cases from the first post, by end of day. Fork it, submit a PR with your edge cases, and I'll merge them into the test suite – your eval criteria will be especially useful because you've already thought about what "success" means for each one.
The fourth article
You're right – "why fixed audits fail" is its own post. I have the logs, and they show that uniform sampling missed 3 out of 4 directional failures because those failures were rare but catastrophic. I'm already sketching the outline:
Fixed sampling: feels fair, fails at the long tail
Adaptive sampling by confidence: catches structured wrongness
Cost‑aware sampling: weight by potential impact, not by random draw
The residual: when confidence and impact diverge (the hardest case)
If I write it, I'll be citing your "most confident when wrong in a structured way" line. Let me know if you want early access to the draft when it's ready.
Thanks for pushing beyond "good comment" into actual contribution. You're making this series better than I could alone.
This is a clean write-up, and the precision-recall framing is the right one. The piece I would add: G4 is being scored as a model-size result when it is really a routing result.
"exit 0, 0 passed (no tests collected)" is not a semantic judgment, it is a parseable fact. You do not need a hundreds-of-B model to catch it, you need to read the collected and passed counts off the summary line. The weak models failed G4 because a structured fact got handed to a fuzzy judge, and the strong model gets it but pays for that reading comprehension with over-rejection everywhere else. So the fix is not a bigger inspector, it is splitting the job: deterministic assertions own everything mechanically checkable (tests collected greater than zero, schema conformance, non-emptiness thresholds), and the LLM only judges the irreducibly fuzzy residue, is this draft substantively adequate. Then the 75% false-reject rate is not applied across all outputs, only across the genuinely ambiguous slice, which is much smaller.
The other thing I would measure differently: in an agent loop a false reject and a false accept do not cost the same, so counting them side by side undersells it. A false accept ships once. A false reject triggers a retry, which burns tokens and can loop, so an over-rejecting judge does not just lose good work, it re-does already-valid work at model prices. When you pick a point on that curve you have to price both errors as they land in the loop, not just tally them.
Last small lever: a lot of the over-rejection is coming from the binary instruction itself. "The output must clearly demonstrate it meets the requirement" turns "I cannot tell" into "reject." Giving the judge an explicit third option, abstain and route to a cheaper deterministic check or a human, keeps the ambiguous cases from all defaulting to a rejection.
Alexey, thank you – this is the most surgical critique I've received. You've dissected exactly the three places where my third‑post harness was still papering over gaps.
Let me go through each point, because they all land on real flaws I found – and some I missed.
G4 as a routing failure, not a model‑size result
You're spot on. The moment I read your comment, I realized: the weak models failed G4 because I fed a parseable fact to a fuzzy judge. The strong model gets it, but pays for that comprehension by over‑rejecting everywhere else.
Your fix – deterministic assertions for everything mechanically checkable (test counts, schema, non‑emptiness) – is exactly the layer I omitted. In the harness (third post), I did have a pre‑flight validation step, but it only checked file existence, not content‑shape. So G4 sailed through. I'm now adding a “structured fact extractor” before any LLM judge, so that 0‑passed case gets killed deterministically, and the judge only sees the semantically ambiguous residue.
Cost asymmetry: false accept vs. false reject
This is a point I barely touched, and you've sharpened it correctly. In my third‑post logs, I saw a false reject trigger up to three retries before the harness gave up – that burned ~3× the token cost of a single pass, plus latency. And if the retry loop doesn't exit cleanly, it can spiral.
So the confusion matrix cannot be symmetric. I'm now planning to assign a multiplicative cost weight to false rejections (say, 3× or 5× of a false acceptance) when tuning thresholds. That flips the optimal point on the precision‑recall curve – we'd rather let a few garbage outputs through than reject a ton of valid work, because in many production contexts, a human can spot a bad output much cheaper than re‑running an agent from scratch.
The third option: abstain → escalate
Your suggestion of an explicit “abstain / route to deterministic check or human” was actually in my harness design – but it failed in practice. I gave the judge a third output ("uncertain"), and the strong model used it as a backdoor: it abstained on ~40% of valid cases, essentially punting all hard decisions to the fallback, which defeated the purpose.
What I learned (and what you hinted at) is that abstention must be constrained – the model must quote the exact failing evidence, and it cannot abstain unless the evidence is unambiguously absent. I'm currently re‑working the prompt to demand a citation and only allow “cannot verify” when the output lacks any verifiable claim. That's closer to your “quote the exact failing evidence” rule, which I fully endorse.
The meta‑question: who guards the guard's premise?
You raised the hardest question: the expectation (e.g., “this task should collect N>0 tests”) is itself often agent‑generated, so it can be gamed. In my harness, I pinned the expectation in a task contract that the executing agent cannot modify – it's stored separately in the orchestration layer. That's the only way I've found to make expectation‑diffing meaningful. But the contract itself is still written by a human or a higher‑level planner, so the chain of trust isn't infinite – it ends at a human or a verified specification.
Your comment effectively redraws my architecture:
Layer 0: deterministic shape/parse checks (kills G4, duck, period, TODO at zero semantic cost).
Layer 1: LLM judge with a constrained abstain option, applied only to the remaining fuzzy cases.
Layer 2: human escalation for the small residual that the judge cannot resolve with high confidence.
I'm already coding this into the harness v2. And I'll make sure the confusion matrix in my next report shows weighted costs, not just raw counts.
Thank you – you turned a set of data points into a clear design pattern. I owe you a citation in the next iteration. If you ever want to co‑design that layered pipeline, I'd be glad to open the repo for collaboration.
Feel free to adjust the tone – you can make it slightly shorter or more casual. This version acknowledges his deep points, references your third post implicitly, and shows you're already acting on his feedback, which is exactly the kind of response a technical reviewer appreciates.
A stronger model reduced false positives (letting bad work pass) but significantly increased false rejections (rejecting good work).
Exactly right — that's the cleanest summary of the Part 2 finding. Stronger models filter garbage better, but they also reject valid work at higher rates.
The interesting property is that this tradeoff appears structural, not tunable. Prompt adjustments (strict vs lenient) shift the operating point along the curve, but they don't break out of it. The root cause is that "does\ this output satisfy the requirement?" is genuinely underspecified for many real agent outputs. A model that reads deeper into the semantic gap will flag ambiguity that a shallower model glosses over —which is correct behavior for catching garbage, but produces false rejections on valid-but-imperfect outputs.
One direction this points toward: instead of optimizing the single LLM judge, route the easy cases away from it. Catch obvious garbage with patterns and obvious passes with length/format checks. The LLM then only sees the residual where the tradeoff is inherent —and those are cases that would need human review either way.