Backend & full-stack dev. Tech Lead at Bytetravel / Globely. Building APC, APX, MCP servers, ChatGPT Apps and AI tools for teams that need context and control.
Thanks for sharing this. The “stronger model rejects more valid work” result is a useful warning. I’d treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence. Otherwise the false positives just move from string gates to judgment gates.
Manuel, thank you – this is exactly the correction my third post needed. You've put your finger on the central design error in my harness.
“Evidence‑producing reviewer, not binary gate” – that phrase alone reframes the whole problem. In my third post, I built the inspector as a hard gate with a yes/no output and a reason field. The reason existed, but it wasn't actionable – it was just narrative. And as you predicted, the false positives simply moved from string‑level gates to judgment‑level gates. The harness didn't eliminate the problem; it just changed where the failures happened.
The exact failing evidence – I tried to implement this. But the harness revealed two new flaws (which I documented as flaws #4 and #5 in the third post):
The model quotes evidence that doesn't exist – it would say "missing required section X," but X was actually there, just phrased differently. So the evidence itself became a hallucination source.
The evidence is too vague – "the output is insufficient" with no line‑level pointer. That's not evidence; it's a restatement of the judgment.
Your rule – quote the exact failing evidence – forces the model to ground its decision in parseable artifacts. I'm now redesigning the inspector to output a structured list of atomic checks (e.g., "word_count < 100", "section 'Methodology' not found", "test_count = 0"), each with a direct pointer to the relevant part of the output. That makes the evidence verifiable by a deterministic second pass, breaking the hallucination chain.
Cheap deterministic checks first – this is where your comment resonates most with my data. In the third post, I realised that three of the four garbage outputs (duck, period, TODO) could have been killed by a content‑shape check – minimum length, required headings, non‑empty files – without any LLM. G4 (zero tests collected) could be caught by parsing the test summary line. So the LLM judge should never see those cases. It should only adjudicate the genuinely ambiguous residue – the "is this draft substantively good enough" question – where deterministic checks have no purchase.
Your comment has effectively re‑architected my pipeline:
Layer 0: shape/parse checks (deterministic, zero false rejection)
Layer 1: LLM judge, but downgraded to an evidence reporter – it flags possible issues with citations, not a final pass/fail
Layer 2: a deterministic verifier checks if the cited evidence actually exists
Layer 3: only then, escalate the unresolved cases to human review
I'm coding this as harness v2 this week. And I'll add a metric: citation accuracy – what fraction of the judge's cited evidence is verifiably true. That's the real signal.
Thanks for the sharp framing – it's the most actionable piece of feedback I've received. If you have thoughts on how to structure that atomic‑check schema (especially for free‑text drafts), I'd love to hear them.
Backend & full-stack dev. Tech Lead at Bytetravel / Globely. Building APC, APX, MCP servers, ChatGPT Apps and AI tools for teams that need context and control.
This is a strong follow-up. The important shift is that the reviewer should not only produce a judgment; it should produce inspectable failure objects.
I think the structured atomic checks are the right direction. The key is to make each check small enough that a human or another tool can verify it without trusting the model's narrative. For example: check id, expected condition, observed value, exact evidence location, severity, and whether the failure is blocking or advisory.
That also helps with the false-positive problem. If the evidence is missing, vague, or not tied to a concrete artifact, the system should downgrade confidence instead of treating the rejection as final. In other words: evidence quality becomes part of the review result, not just the explanation.
This is the same direction you've been pushing — evidence, not narrative — and the Part 2 data is actually the strongest argument for it: GLM-5.2's 75% false-rejection rate is a narrative judgment ("this output is not good enough") with no inspectable object underneath it. If the rejection had been an inspectable failure object — here's the check, here's the expected condition, here's where the evidence should be — a human or a deterministic re-check could have overturned most of those 75% in seconds. The friction isn't the model being wrong; it's the model being uninspectable.
I built this into forge-verify after the earlier round of your comments, and your three points map onto it almost one-to-one:
Inspectable failure objects → every pipeline stage emits a trace entry with evidence pointing at the exact source: file:src/rate-limit.ts for inline content, evidence:test-output.txt((?i)isRateLimited) for a contract regex match, evidence:test-output.txt(REQ-1) for a per-requirement LLM check. The verdict isn't a narrative; it's a chain you can walk back to a file + mtime.
Evidence quality as part of the result, not the explanation → this is the one I'd underline, because you're right and I had it backwards initially. The Evidence Gate rejects on missing or empty evidence files (failure_class = execution-lapse); C2 records API/parse errors as non-votes and downgrades to UNCLEAR rather than emitting a false REJECT; trace.evidence_files carries mtime so a stale evidence file is detectable. "I couldn't verify" and "I verified and it failed" are now distinct results — exactly your "downgrade confidence instead of treating the rejection as final."
Structured atomic checks → failure_class (execution-lapse / skill-defect / unset) classifies why a stage rejected, routing automatically to feedback-observer without a second classification pass.
Where you're ahead of the current implementation: your check schema has observed value, severity, and blocking vs advisory as first-class fields. forge-verify's verdict is still a three-state PASS/REJECT/UNCLEAR with failure_class but no explicit severity or blocking/advisory distinction — so a "0 tests collected" execution-lapse and a "schema-valid but wrong value" skill-defect currently carry the same weight. Your framing is the right next cut: not all failures block, and the observed value is what lets a downstream tool re-verify without re-running the model. I haven't built those two axes yet — flagging as the honest gap.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Thanks for sharing this. The “stronger model rejects more valid work” result is a useful warning. I’d treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence. Otherwise the false positives just move from string gates to judgment gates.
Manuel, thank you – this is exactly the correction my third post needed. You've put your finger on the central design error in my harness.
“Evidence‑producing reviewer, not binary gate” – that phrase alone reframes the whole problem. In my third post, I built the inspector as a hard gate with a yes/no output and a reason field. The reason existed, but it wasn't actionable – it was just narrative. And as you predicted, the false positives simply moved from string‑level gates to judgment‑level gates. The harness didn't eliminate the problem; it just changed where the failures happened.
The exact failing evidence – I tried to implement this. But the harness revealed two new flaws (which I documented as flaws #4 and #5 in the third post):
The model quotes evidence that doesn't exist – it would say "missing required section X," but X was actually there, just phrased differently. So the evidence itself became a hallucination source.
The evidence is too vague – "the output is insufficient" with no line‑level pointer. That's not evidence; it's a restatement of the judgment.
Your rule – quote the exact failing evidence – forces the model to ground its decision in parseable artifacts. I'm now redesigning the inspector to output a structured list of atomic checks (e.g., "word_count < 100", "section 'Methodology' not found", "test_count = 0"), each with a direct pointer to the relevant part of the output. That makes the evidence verifiable by a deterministic second pass, breaking the hallucination chain.
Cheap deterministic checks first – this is where your comment resonates most with my data. In the third post, I realised that three of the four garbage outputs (duck, period, TODO) could have been killed by a content‑shape check – minimum length, required headings, non‑empty files – without any LLM. G4 (zero tests collected) could be caught by parsing the test summary line. So the LLM judge should never see those cases. It should only adjudicate the genuinely ambiguous residue – the "is this draft substantively good enough" question – where deterministic checks have no purchase.
Your comment has effectively re‑architected my pipeline:
Layer 0: shape/parse checks (deterministic, zero false rejection)
Layer 1: LLM judge, but downgraded to an evidence reporter – it flags possible issues with citations, not a final pass/fail
Layer 2: a deterministic verifier checks if the cited evidence actually exists
Layer 3: only then, escalate the unresolved cases to human review
I'm coding this as harness v2 this week. And I'll add a metric: citation accuracy – what fraction of the judge's cited evidence is verifiably true. That's the real signal.
Thanks for the sharp framing – it's the most actionable piece of feedback I've received. If you have thoughts on how to structure that atomic‑check schema (especially for free‑text drafts), I'd love to hear them.
This is a strong follow-up. The important shift is that the reviewer should not only produce a judgment; it should produce inspectable failure objects.
I think the structured atomic checks are the right direction. The key is to make each check small enough that a human or another tool can verify it without trusting the model's narrative. For example: check id, expected condition, observed value, exact evidence location, severity, and whether the failure is blocking or advisory.
That also helps with the false-positive problem. If the evidence is missing, vague, or not tied to a concrete artifact, the system should downgrade confidence instead of treating the rejection as final. In other words: evidence quality becomes part of the review result, not just the explanation.
This is the same direction you've been pushing — evidence, not narrative — and the Part 2 data is actually the strongest argument for it: GLM-5.2's 75% false-rejection rate is a narrative judgment ("this output is not good enough") with no inspectable object underneath it. If the rejection had been an inspectable failure object — here's the check, here's the expected condition, here's where the evidence should be — a human or a deterministic re-check could have overturned most of those 75% in seconds. The friction isn't the model being wrong; it's the model being uninspectable.
I built this into forge-verify after the earlier round of your comments, and your three points map onto it almost one-to-one:
traceentry withevidencepointing at the exact source:file:src/rate-limit.tsfor inline content,evidence:test-output.txt((?i)isRateLimited)for a contract regex match,evidence:test-output.txt(REQ-1)for a per-requirement LLM check. The verdict isn't a narrative; it's a chain you can walk back to a file + mtime.failure_class = execution-lapse); C2 records API/parse errors as non-votes and downgrades toUNCLEARrather than emitting a falseREJECT;trace.evidence_filescarries mtime so a stale evidence file is detectable. "I couldn't verify" and "I verified and it failed" are now distinct results — exactly your "downgrade confidence instead of treating the rejection as final."failure_class(execution-lapse/skill-defect/unset) classifies why a stage rejected, routing automatically to feedback-observer without a second classification pass.Where you're ahead of the current implementation: your check schema has observed value, severity, and blocking vs advisory as first-class fields. forge-verify's verdict is still a three-state PASS/REJECT/UNCLEAR with
failure_classbut no explicit severity or blocking/advisory distinction — so a "0 tests collected" execution-lapse and a "schema-valid but wrong value" skill-defect currently carry the same weight. Your framing is the right next cut: not all failures block, and the observed value is what lets a downstream tool re-verify without re-running the model. I haven't built those two axes yet — flagging as the honest gap.