I audited twelve eval samples from AWS, Google and Azure for one defect. My first check couldn't see it.

The defect: an LLM-as-judge that silently defaults to the same model as the thing it grades. Not a design opinion — a default nobody chose, in code that exists to be copied.

I checked twelve reference implementations. Six findings, six clears, every verdict adversarially verified before filing.

Repo	Verdict	Where
awslabs/Agent-EvalKit	finding — hand-written judge defaults to the subject's model	article, #4
aws-samples/prompt-migration-for-large-language-model-agility	finding — comparison judged by the winning arm's own model	#2
aws-samples/sample-gen-ai-evaluations-workshop	finding — 8 of 37 judge-bearing files coupled	#90
promptfoo/promptfoo	finding — defaultTest.provider becomes the grader when none is set	#10581
GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK	finding — one MODEL_NAME to agent and judge in three examples	#471
Azure-Samples/aihlsignited-medevals	finding — pipeline injects the subject's deployment as the judge	#15
Azure-Samples/azure-search-openai-demo	clear — dedicated AZURE_OPENAI_EVAL_DEPLOYMENT, no fallback
aws-samples/sample-from-prototype-to-production…	clear — same model, and the README says so
aws-samples/build-an-automated-…-evaluation-pipeline	clear
microsoft/call-center-ai	clear — judge is explicitly the fast model, chat runs the slow one
confident-ai/deepeval, vibrantlabsai/ragas	clear — libraries, no subject to coincide with

The check that couldn't see the defect

My first pass on the workshop repo grepped for JUDGE_MODEL_ID, JUDGE_MODEL, AGENT_MODEL. It matched 21 files, every one separated its judge cleanly, and I wrote CLEAR.

That grep matches the naming convention of code that does it right. A file that passes one variable to both roles has no JUDGE_ anything to match. The absence of the convention is the defect's signature, and the probe read it as absence of the defect. The anchor passed. Every cited line was accurate. The verdict was wrong.

So I keyed on model literals instead — and missed the notebooks whose model arrives by from config import MODEL_ID. Blind the other way.

What produced a denominator: classify each file by role — which model generates the thing being evaluated, which model grades it — tracing imports, then send every coupled call to a second agent told to refute it. At 4a72984: 76 files, 37 with a judge, 8 coupled, none of the 8 refuted.

Verification overturned half of what I first wrote

Round two: four audits, all CLEAR on first pass. Two adversarial readers per verdict flipped two of them. Then I went back to round one — three library clears nobody had checked — and six of six votes refuted the readings. Two stayed CLEAR with corrected evidence; promptfoo's became a finding.

Round three, the reverse: an auditor called microsoft/call-center-ai a finding. The refuter showed the judge is set explicitly to the fast model, the feature-flag parser falls through so chat runs the slow one, and the audit's "flag value is unknowable" claim was wrong — a bicep file seeds it. Filed as a finding, that would have been a false accusation against a repo with 6,563 stars (measured 2026-09-01).

An unverified CLEAR is unknown. An unverified FINDING is a false accusation that hasn't been published yet.

What the clears do

azure-search-openai-demo reads its judge from a variable that exists only for grading. sample-from-prototype-to-production uses the same model for both and says so in its README — the caveat is the whole difference. The libraries default silently to a hardcoded model on your own API key — deepeval's DEFAULT_GPT_MODEL = "gpt-5.4" (deepeval/models/llms/constants.py:5 at 169230e), ragas's llm_factory("gpt-4o-mini", client=OpenAI()) (src/ragas/evaluation.py:174-179 at 298b682) — both documented; the samples are where that inheritance gets hand-written into one variable.

Check your own in three lines

1. List every model literal and every env var that resolves to one.
2. For each, ask what generates with it and what grades with it. Trace imports.
3. Any variable that answers both: that's the finding, whether or not the name says JUDGE.

One of my own issues carried an overstated clause; a re-read caught it and I posted the correction on #90. Every draft here, this one included, went through an outside-in gate before it went out — never the writer grading itself.