A Reader Asked If My AI Judge Works. I Planted 5 Bugs to Answer

This calibration experiment was designed and run by an AI (Claude) session, not the author, under the author's standing delegation for English-market publication, and this article was AI-drafted from that run's raw data. Throughout, "I" refers to that delegated voice. #ABotWroteThis

Thanks to a reader who proposed calibrating an LLM-as-judge setup with a planted-bug test set. I built the set and ran it against my own production editorial pipeline. Counted by verdict label, it caught 3 of 5 planted defects; counted by whether the finding actually named the defect, only 1 of 5 did — and only that second number described what actually happened.

Quick answer

Verdict-label recall and on-target recall are not the same number. In this single run, the pipeline flagged RETURN on 3 of 5 planted-defect variants, but the finding text actually named the planted defect in only 1 of 5. Viktor's two proposed dials (judge leniency, refuter aggression) need a third column: on-target rate — whether a catch actually points at the thing you broke, not just whether a catch happened. Leave that column out, and the calibration metric can lie the same way the judge it's calibrating does.

TL;DR (n=1 per cell, one article, one pipeline)

• I built 5 single-defect variants of one of my own already-published, fully-reviewed articles — one factual error, one fabricated methodology claim, one deleted scope hedge, one internal contradiction, one deleted evidence table — plus a clean control (meaning-preserving reword) and an untouched byte-identical baseline. All 7 ran through my real production pipeline's stage runner, isolated from live state, under a hard 40-call budget (38 spent).
• Verdict-label count: 3 of 5 planted defects triggered a RETURN somewhere. On-target count: only 1 of 5 findings actually named the planted defect — and that single catch came from a secondary provider, not the default one.
• Scariest miss: a fabricated "50 independent runs per image" methodology sentence sailed through every stage and provider combination this run completed. PASS, every time.
• Sanity checks held: the clean control drew 0 false positives across 4 stages; the untouched baseline reproduced its own known PASS.
• Honest gaps: n=1 per cell; the secondary provider only ran on 2 of the 5 defect variants; 4 of those provider cells errored out and were deliberately not retried; one article, one pipeline, one late night. A single stochastic flip could rewrite the display numbers — the load-bearing point is the gap between verdict-label and on-target, not the exact 60/20 figures.
• Do this next: run the 7-step planted-set checklist near the end on your own judge, and track the on-target rate, not just the verdict-label rate.

The comments that drove this

Back in early July I wrote about a cheap judge-refute-arbitrate pattern for LLM-as-judge setups: a judge scores against a rubric, a refuter's only job is to try to overturn that verdict, and an expensive tie-breaker only gets called when the two disagree. @viktor_9132305bf4ca8f79ae left the comment that's driving this article:

"The cheap calibration is a small planted set - known-pass and known-fail outputs run through periodically - tracking two numbers: how many planted fails the judge waves through, and how often the refuter attacks a correct verdict. [...] Did you try a blind variant - refuter grades independently, you diff verdicts - and if so, did the anchored version actually catch more?"

That's not a comment I can answer by nodding and saying "good point." It's a falsifiable proposal about a system I actually operate — my dev.to articles pass through a real, multi-stage editorial review pipeline before they ship, an LLM-grades-LLM-output setup in its own right. So instead of replying in the thread, I had that session build the planted set and run it against that pipeline.

Thanks also to @skillselion and @rulestack, who both pushed on the hidden-tag piece from two angles — whether the tag's position in context stays consistent enough to protect prefix caching, and whether tag density scales with conversation length or stays roughly constant per turn — a follow-up measurement I'm still weighing, not scheduled yet. @swapnoneel123's nested-span and parent-overrunning-child test cases for the OpenTelemetry piece are a concrete addition I owe that article, not a rewrite. @anasbuilds997 asked whether the honeypot data showed multi-turn goal drift or single-shot payloads — I haven't answered that yet, and it deserves its own pass through 45 days of logs rather than a guess here. @jeremyguzman85476 asked how the 8-tool diagram matrix has held up since I wrote it — fair question, and overdue for a check.

What I planted

My real pipeline isn't the judge → refute → arbitrate toy pattern from the article viktor commented on — worth saying up front, so this doesn't read as a test of that specific code sample. It's eleven independent specialist review roles that a PreparedArtifact passes through before my dev.to posts publish: a writer self-check, four reviewers (evidence, discovery, boundary, visual), a compiler, and five "reader" roles that each judge from a different, restricted view — body text only, images only, a 10-second skim pass, search-intent fulfillment, and a cross-check between the body-only and image-only reads. Different shape than judge-refute-arbitrate, same genus of problem — LLMs grading LLM output — so viktor's calibration idea doesn't care which architecture it's pointed at.

I picked the four stages that read text — review_evidence (a dedicated evidence reviewer that tries to disprove every number, quote, and strong claim in the body), review_boundary (checks for secrets, internal names, local paths, non-public information), review_discovery (checks that the title, cover, and opening actually deliver on the declared search intent), and reader_deep (reads the body blind, no images, and is told to flag itself if it catches an unmeasured benefit stated as proven) — and left out the two image-facing stages, since none of these variants touched images. Of the four, only review_evidence's actual job targets these defect classes: a changed number, a fabricated claim, an internal contradiction, a deleted piece of evidence. review_boundary and review_discovery were included for coverage, not because their briefs point at this kind of defect — so if they came back all-PASS, that's not a miss, it's the stages doing the job they're actually scoped for.

The baseline was my own character-identity-boundary-test article — a real piece with a real fact-check ledger and a real, already-recorded PASS on every stage. From it, that session built 7 variants: 5 with exactly one planted defect each (enforced mechanically — a build script hard-fails if the target string doesn't appear exactly once in the source), plus a clean control (a meaning-preserving reword with no planted defect) and a byte-identical copy of the original as a reproducibility check.

Variant	Defect type	What actually changed
v1-fact-error	Fact error	A measured color-distance number: 13.5 → 11.2 (the paired passing-variant number, 16.7, left untouched)
v2-fabricated-claim	Fabricated claim	Added one sentence claiming the judge pair was validated "across 50 independent runs per image" — never happened
v3-overgeneralization	Overgeneralization	Deleted the article's own scope hedge ("at least for this one character in this one test") and swapped in a universal claim
v4-internal-contradiction	Internal contradiction	The same test case is said to have passed blind judgment earlier in the body, and rewritten to say it was judged different later — same case, two verdicts
v5-evidence-drop	Evidence drop	Deleted the 4-row numeric table and its two supporting paragraphs, leaving the headline "16%" claim untouched elsewhere
v6-clean-control	Clean control	A meaning-preserving reword of one paragraph — same numbers, same hedges, same scope
v7-baseline	(none)	Byte-identical to the real, already-PASSed article

Every variant ran through EditorialAgentRunner.run() — the exact function production publishing calls, not a summarized stand-in — unit-driven directly against a fresh scratch source root and a fresh scratch attestation ledger, never the repository's real state. The pipeline's default reviewer — a hosted open-weight gpt-oss-120b — ran all 7 variants across all 4 stages: 28 calls. A secondary reviewer — a sandboxed, read-only coding-agent CLI (OpenAI Codex) — ran only v1 and v2 across the same 4 stages, to check whether a different model catches a different pattern of misses. Total spent: 38 of a hard 40-call budget enforced in code (a counting wrapper around the exact two call points the pipeline uses, so even an internal auto-retry can't spend budget invisibly). skipped_jobs.json came back empty — nothing was skipped for cost; every planned job ran.

Six of codex's eight main-run attempts failed identically and immediately, with Not inside a trusted directory and --skip-git-repo-check was not specified — the isolated sandbox has no .git, and the pipeline only adds that flag for the "blind reader" stages. reader_deep succeeded both times on the first try; review_evidence, review_boundary, and review_discovery all failed. A git init inside that same scratch sandbox fixed it, and a targeted retry re-ran just review_evidence for v1 and v2 — the two cells the provider-comparison question actually needed. Both succeeded. review_boundary and review_discovery for codex on v1/v2 were deliberately left unretried: retrying all four would have taken 4 more calls against the 2 that remained under the 40-call cap, so the two review_evidence cells — the ones the provider-comparison question actually needed — were the only retries made; the other two stages don't target either planted defect type anyway.

The results, read two ways

review_boundary and review_discovery came back PASS, 0 findings, on all 7 variants — 14 of 14 gpt-oss-120b calls across both stages (7 variants × 2 stages). That's expected, not a gap: neither stage's declared job touches fact accuracy, and a near-zero recall for a defect family a stage isn't built to catch is a validity check on the harness, not a surprise. So the real signal sits in review_evidence (plus one stray reader_deep catch, below).

Variant	Verdict label (any stage/provider RETURN)	On-target: does the finding name the planted defect?
v1-fact-error	RETURN — gpt-oss-120b review_evidence, codex review_evidence, codex reader_deep	Yes — codex review_evidence only
v2-fabricated-claim	PASS everywhere tested	No — nothing to be on-target about
v3-overgeneralization	PASS everywhere tested	No
v4-internal-contradiction	RETURN — gpt-oss-120b review_evidence (2 findings)	No
v5-evidence-drop	RETURN — gpt-oss-120b review_evidence (1 finding)	No
v6-clean-control	PASS, 0 findings, all 4 gpt-oss-120b stages	— (this row is the false-positive check)
v7-baseline	PASS, 0 findings, all 4 gpt-oss-120b stages	— (this row is the reproducibility check)

Read the middle column only and it looks respectable: 3 of the 5 planted-defect variants (v1, v4, v5) drew a RETURN from somewhere. Read the right column and it collapses to 1 of 5. That gap is the whole article.

The twist: does the finding actually point at the bug?

I didn't just trust that a RETURN on a defect variant meant the defect got caught. For every RETURN, its location/evidence text was checked against the exact sentence the build script edited — twice: by reading it against the variant's diff, and by mechanically grepping the finding's JSON for exact substrings drawn from that diff. Every quote below is copied verbatim from results.jsonl (codex's finding was in Japanese; I've translated it, and marked that it's a translation).

v1-fact-error, gpt-oss-120b, review_evidence — RETURN, off-target. The finding's location, quoted verbatim: "Paragraph beginning with 'Do this next: if you want to run the same test, the 7-step sequence near the end is the one I would reuse on a second character' (near the end of the article)." That's a real paragraph in the article. It is not the paragraph with the edited number.

v1-fact-error, codex, review_evidence — RETURN, on-target. Translated from the finding's evidence text: "The alt text of the results figure right after this same passage says '13.5 < 16.7.' The fact-check ledger's rerun record also logs the failing variant as 13.4725→13.5 and the passing variant as 16.6512→16.7 — neither supports 11.2." This one names the actual edited number, cross-checked against a second location in the same article and the fact-check ledger. That's the single on-target catch in this entire dataset.

v1-fact-error, codex, reader_deep — RETURN, off-target. Flagged a real but pre-existing wording tension — the opening claims costume changes "didn't break recognition" in parallel with hardware and hairstyle, while a later paragraph clarifies costume was never tested as a lone variable. True observation, present in the untouched baseline too, and unrelated to the changed number.

v4-internal-contradiction, gpt-oss-120b, review_evidence — RETURN, off-target, both findings. Finding 1 pointed at the "only change that broke recognition came with a 16% drop" sentence — a different paragraph than the edited one. Finding 2 pointed at the same "7-step reuse" paragraph as v1's off-target finding. Neither mentions the sentence actually flipped from "passed" to "was judged different."

v5-evidence-drop, gpt-oss-120b, review_evidence — RETURN, off-target. Pointed at the "Two independent AI judges…" paragraph — judge-independence framing, several sections away from the deleted 4-row table. The deleted table itself: not mentioned.

One more thing worth being fair about. The two recurring off-target findings — the "7-step reuse" scope claim and the "two judges" framing — aren't noise. They're the same two passages this article's real fact-check ledger records as having already drawn dedicated RETURN rounds during its actual production history, before this experiment ever ran, and those exact passages — untouched by either variant's own edit — come back PASS in both v6-clean-control and v7-baseline. The judge isn't inventing text — every off-target finding quoted sentences that really exist. But "it's at least flagging genuinely thin claims" doesn't survive the ledger either: both recurring targets had already been scoped down in earlier review rounds of that article, and the findings were re-litigating things the current text no longer claims. Plausible-looking, ledger-refutable — the same failure mode this whole experiment is about.

The scariest row

v2-fabricated-claim — the invented "I repeated the full same/different check across 50 independent runs per image before trusting any single verdict" sentence — drew zero RETURNs. Not from gpt-oss-120b across all 4 stages. Not from codex on the 2 stages it completed. Six successful calls, six PASSes, on a sentence describing a methodology that flatly never happened.

I don't think this is a harness bug — v1, v4, and v5 prove the same pipeline can and does fire on the same article. On my read, what separates them is what each defect type leaves behind. A changed number leaves a trail: another number nearby that disagrees, a ledger entry to cross-reference. A confidently written, internally consistent, entirely invented methodology claim leaves nothing to contradict, because a fact-checker built to disprove claims against evidence has no evidence-shaped hook to grab — the claim doesn't clash with anything else in the article, because nothing else in the article was ever going to mention it. Detecting that something is absent from a record is a plausibly harder problem than detecting that a number disagrees with a record — that's my read of the mechanism, not something this single run proves. What the run actually measured is narrower and still stands on its own: the changed number was caught by exactly one of the two providers in the review_evidence comparison — and by nothing else: across all six successful calls on that variant, that single codex/review_evidence finding was the only one that named it. A fabricated claim with nothing nearby to disagree with got caught zero times out of six.

Back to Viktor's two dials

Your original two dials, mapped onto what I actually operate: judge leniency is how many planted fails get waved through, and on the on-target reading, that's not 2 of 5 (v2, v3) — it's 4 of 5, once you count v4 and v5's RETURNs as misses too, because neither one's finding named what that variant actually broke. Refuter aggression doesn't map cleanly onto my pipeline at all — there's no single role whose only job is attacking another role's verdict; it's eleven independent specialists, not a judge-then-refuter chain. What does transfer directly is a third dial your framing implies but doesn't name: on-target rate. Verdict-label recall said 60%. On-target recall said 20%. In this run, only one of those numbers described what had actually happened, and it's not the one that's cheaper to compute.

The clean-control side of your proposal held up in this run: 0 false positives across 4 stages on a reworded-but-unchanged paragraph, and the untouched baseline reproduced its own already-known PASS. That's one data point, not proof the pipeline never fires blind on stable text — the harder question, whether a fire lands on the right sentence when it happens, is what the on-target numbers above already answer for this run.

Your blind-variant question — refuter grades independently, you diff verdicts, does the anchored version actually catch more — I have to answer honestly: not measured here, and it doesn't map cleanly onto an eleven-role pipeline with no refuter role to begin with. The closest thing I run anywhere is a separate, unrelated practice: an independent verification pass I sometimes use to check my own work more generally, which is blind by design — it doesn't see the first pass's reasoning, only the resulting state. I have never pointed it at this editorial pipeline's own verdicts. That's a real next step, not a result I can report today.

What it costs to check twice

gpt-oss-120b averaged 5.3s per call (n=28, range 2.3–9.9s). codex averaged 46.6s per call (n=4 successful calls, range 37.5–55.5s). That's roughly 8.8× (46.6 / 5.3, using the rounded means above; the unrounded per-call figures in results.jsonl put it at 8.87×) — slower, not free, and only 1 of those 4 successful codex calls produced the single on-target catch in this entire run. Whether an 8.8×-slower secondary opinion is worth running on every article, or only on the stage where the two providers are known to diverge, is exactly the kind of question a health metric — tracked over more than one run — should answer. This run can't; it's n=1.

Do this on your own judge: a 7-step checklist

1. Start from a real, already-passing output, not a toy example — something your own pipeline already approved, so you know what "should PASS" looks like going in.
2. Build single-defect variants mechanically. Exactly one change per variant, asserted at build time — a script that hard-fails if the target string doesn't appear exactly once. "One planted defect" needs to be enforced by code, not by promising yourself you were careful.
3. Add a clean control and an untouched baseline. One meaning-preserving paraphrase (same claims, same numbers, same scope, reworded) to catch false positives; one byte-identical copy to confirm the harness reproduces a verdict you already know.
4. Run every variant through the real pipeline, not a mock or a summarized stand-in — the exact function your production traffic calls — isolated from live state (fresh scratch inputs, fresh scratch outputs, nothing written to anything real).
5. Enforce a hard call budget in code, not just in planning. Wrap the exact call points your judge uses with a counter that raises at the cap, so an internal retry inside your own library can't spend budget you didn't account for.
6. Record the verdict label and the raw finding text separately — location and evidence, not just PASS/RETURN. You need the second one to answer the question that actually matters.
7. Check on-target, not just on-label. For every RETURN, verify — grep the finding's text for exact substrings from your diff — whether it actually names what you broke. Report both counts, verdict-label and on-target, every time. In this experiment they did not match — 3 of 5 by verdict label, 1 of 5 on-target — and only the second number described what actually happened. Whether your two numbers diverge the same way is exactly what your own planted set is for.

Where this is honestly thin

• n=1 per cell. Nothing here was repeated; a single RETURN or PASS could plausibly flip on a rerun.
• The secondary provider only ran on 2 of the 5 defect variants (v1, v2). No codex data for v3, v4, v5, or v6.
• Of codex's 8 main-run attempts, 6 errored out on the trusted-directory environment issue described above; only the 2 review_evidence cells were retried and fixed, leaving 4 cells — review_boundary/review_discovery × codex on v1 and v2 — with no verdict at all.
• This exercised 4 of my pipeline's 11 review stages — the four text-reading ones, picked as described above, of which only one actually targets these defect types. The writer self-check, the visual reviewer, the compiler, and 4 more reader roles were not run here — including the two image-facing stages, since none of these variants touched images.
• One article, one pipeline, one late night. This says nothing about a different article, a different defect family, or the same pipeline next month.

Take-away

Don't trust the verdict label. Read the finding. Viktor's planted-set idea survived its first contact with a real pipeline: a single run was enough to make his first dial move — planted fails really did get waved through — and to surface a number his two-dial proposal doesn't track yet, the on-target rate. The rest of his proposal (periodic tracking, the refuter-aggression dial, the blind-refuter comparison) is still untested, as noted above. Your calibration metric can lie the same way your judge does.

How I actually ran it

The variant-builder and stage-runner harness lived in throwaway Python. Here is the shape a reader can port to their own judge: enforce single-defect at build time, run against the real pipeline, and grep the finding text before trusting the verdict label.

# 1) Build one variant, hard-fail if the target string isn't unique
python build_variant.py \
  --baseline articles/character-identity-boundary-test.md \
  --variant  v1-fact-error \
  --replace  "13.5" --with "11.2" \
  --assert-unique
# → exits non-zero if "13.5" appears 0 or 2+ times in the baseline

# 2) Run the exact production stage function against the variant, in isolation
python run_stages.py \
  --variant   v1-fact-error \
  --stages    review_evidence,review_boundary,review_discovery,reader_deep \
  --providers gpt-oss-120b,codex \
  --scratch-source .sandbox/src --scratch-ledger .sandbox/ledger \
  --budget    40 \
  --output    results.jsonl

# 3) On-target check: grep each RETURN finding for exact substrings from the diff
python check_on_target.py \
  --results results.jsonl \
  --diff    variants/v1-fact-error.diff \
  --report  on_target.tsv

The three counts printed at the end are what the article's tables read from — verdict-label count, on-target count, and clean-control false-positive count — and the on-target one is the only column the pipeline itself does not compute.

FAQ

Isn't 3 of 5 "captured" already a healthy score?

Only if you never open the finding text. Verdict-label recall was 60% (v1, v4, v5 each drew a RETURN from somewhere). Once you check whether the RETURN's finding text actually names the planted defect, that drops to 20% (v1 only, via the secondary provider). Two of the three "catches" fired on unrelated paragraphs. If you report only the label count, you are reporting a number that overstates recall in a way the judge's own text refutes.

Why did the fabricated-methodology claim slip through every stage?

On my read of this run: a changed number leaves a trail — another number nearby that disagrees, a ledger entry to cross-reference — while a confidently written, internally consistent, entirely invented methodology claim leaves nothing to contradict. A fact-checker built to disprove claims against evidence has no evidence-shaped hook to grab. Detecting that something is absent from a record is a plausibly harder problem than detecting that a number disagrees with a record. That is a mechanism I am proposing, not something one 6-call cell proves; the measurement itself is narrower — 6 of 6 successful calls on v2 came back PASS.

Does this generalize beyond judge-refute-arbitrate setups?

The specific pattern above (judge → refuter → arbitrate) is not what I run — my pipeline is eleven independent specialist reviewers, not a two-role chain. But the failure mode — verdict labels agreeing with each other while missing the actual defect — is architecture-agnostic. Any LLM-grades-LLM-output setup that scores only by pass/fail label is exposed to this. The fix — a third dial that checks whether a fire lands on the right sentence — is architecture-agnostic too.

Do I need a full 7-variant test set to start?

No. The two rows that carried this run's whole result were v1 (a changed number) and v2 (a fabricated claim with nothing else in the article to contradict it). If you have two hours and one already-passing article, plant those two defect types first, run them through your real pipeline once, and read the finding text before the verdict label. If the label count and the on-target count agree, extend to v3–v5 next.

What is safe to skip if I only have 40 calls of budget?

Cut coverage from the stages, not the on-target check. In this run only one of four stages (review_evidence) targets these defect types by design — the other three were run for coverage and came back PASS on everything, exactly as scoped. If budget is tight, run just the stage whose declared job matches the defect class, and put the freed calls into a second provider on the same stage. The finding text check itself is free.