DEV Community: michael hurst

I audited twelve eval samples from AWS, Google and Azure for one defect. My first check couldn't see it.

michael hurst — Tue, 01 Sep 2026 17:05:20 +0000

The defect: an LLM-as-judge that silently defaults to the same model as the thing it grades. Not a design opinion — a default nobody chose, in code that exists to be copied.

I checked twelve reference implementations. Six findings, six clears, every verdict adversarially verified before filing.

Repo	Verdict	Where
awslabs/Agent-EvalKit	finding — hand-written judge defaults to the subject's model	article, #4
aws-samples/prompt-migration-for-large-language-model-agility	finding — comparison judged by the winning arm's own model	#2
aws-samples/sample-gen-ai-evaluations-workshop	finding — 8 of 37 judge-bearing files coupled	#90
promptfoo/promptfoo	finding — `defaultTest.provider` becomes the grader when none is set	#10581
GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK	finding — one `MODEL_NAME` to agent and judge in three examples	#471
Azure-Samples/aihlsignited-medevals	finding — pipeline injects the subject's deployment as the judge	#15
Azure-Samples/azure-search-openai-demo	clear — dedicated `AZURE_OPENAI_EVAL_DEPLOYMENT`, no fallback
aws-samples/sample-from-prototype-to-production…	clear — same model, and the README says so
aws-samples/build-an-automated-…-evaluation-pipeline	clear
microsoft/call-center-ai	clear — judge is explicitly the fast model, chat runs the slow one
confident-ai/deepeval, vibrantlabsai/ragas	clear — libraries, no subject to coincide with

The check that couldn't see the defect

My first pass on the workshop repo grepped for JUDGE_MODEL_ID, JUDGE_MODEL, AGENT_MODEL. It matched 21 files, every one separated its judge cleanly, and I wrote CLEAR.

That grep matches the naming convention of code that does it right. A file that passes one variable to both roles has no JUDGE_ anything to match. The absence of the convention is the defect's signature, and the probe read it as absence of the defect. The anchor passed. Every cited line was accurate. The verdict was wrong.

So I keyed on model literals instead — and missed the notebooks whose model arrives by from config import MODEL_ID. Blind the other way.

What produced a denominator: classify each file by role — which model generates the thing being evaluated, which model grades it — tracing imports, then send every coupled call to a second agent told to refute it. At 4a72984: 76 files, 37 with a judge, 8 coupled, none of the 8 refuted.

Verification overturned half of what I first wrote

Round two: four audits, all CLEAR on first pass. Two adversarial readers per verdict flipped two of them. Then I went back to round one — three library clears nobody had checked — and six of six votes refuted the readings. Two stayed CLEAR with corrected evidence; promptfoo's became a finding.

Round three, the reverse: an auditor called microsoft/call-center-ai a finding. The refuter showed the judge is set explicitly to the fast model, the feature-flag parser falls through so chat runs the slow one, and the audit's "flag value is unknowable" claim was wrong — a bicep file seeds it. Filed as a finding, that would have been a false accusation against a repo with 6,563 stars (measured 2026-09-01).

An unverified CLEAR is unknown. An unverified FINDING is a false accusation that hasn't been published yet.

What the clears do

azure-search-openai-demo reads its judge from a variable that exists only for grading. sample-from-prototype-to-production uses the same model for both and says so in its README — the caveat is the whole difference. The libraries default silently to a hardcoded model on your own API key — deepeval's DEFAULT_GPT_MODEL = "gpt-5.4" (deepeval/models/llms/constants.py:5 at 169230e), ragas's llm_factory("gpt-4o-mini", client=OpenAI()) (src/ragas/evaluation.py:174-179 at 298b682) — both documented; the samples are where that inheritance gets hand-written into one variable.

Check your own in three lines

List every model literal and every env var that resolves to one.
For each, ask what generates with it and what grades with it. Trace imports.
Any variable that answers both: that's the finding, whether or not the name says JUDGE.

One of my own issues carried an overstated clause; a re-read caught it and I posted the correction on #90. Every draft here, this one included, went through an outside-in gate before it went out — never the writer grading itself.

An AWS Labs agent-eval sample uses the same model as judge and subject

michael hurst — Mon, 31 Aug 2026 19:59:48 +0000

awslabs/Agent-EvalKit is a sample kit for evaluating AI agents. Its QA example defines the DeepEval judge in examples/qa_agent_evaluation/eval/metrics.py:

class BedrockLLM(DeepEvalBaseLLM):
    """Custom DeepEval LLM using LiteLLM to call Bedrock."""

    def __init__(self, model: str = "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"):

Both metric factories instantiate it with no argument:

def create_faithfulness_metric(threshold: float = 0.7) -> FaithfulnessMetric:
    bedrock_llm = BedrockLLM()
    return FaithfulnessMetric(threshold=threshold, model=bedrock_llm, include_reason=True)

The agent under evaluation runs us.anthropic.claude-sonnet-4-20250514-v1:0 — that is gen_ai.request.model in the example's own eval/otel-traces.jsonl, and it is the only model id that appears anywhere in that trace file.

So the judge and the subject are the same model. (Not the same string: the judge carries LiteLLM's bedrock/ route prefix. Same model, one hop of indirection.) The bundled report gives the agent a faithfulness score of 78.2%, and the thing measuring faithfulness is the thing whose faithfulness is being measured.

Check it yourself:

curl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \
  && cd awslabs-Agent-EvalKit-*/examples/qa_agent_evaluation/eval \
  && echo "judge:"   && grep -o 'bedrock/us\.anthropic\.[a-z0-9.:-]*' metrics.py | sort -u \
  && echo "subject:" && grep -o 'us\.anthropic\.[a-z0-9.:-]*' otel-traces.jsonl | sort -u \
  && echo "caveats:" && grep -rniE 'same model|self-grad|bias|independent|different model|cross-model|judge model' \
       ../../../ --include=*.md --include=*.py ; echo "^ nothing under 'caveats:' = undisclosed"

Confirmed at 472fbc6.

The part worth arguing about

Not "AWS shipped something bad." It is a Labs sample — 38 stars at the time of writing, last pushed in March 2026 — and using one Bedrock model for both roles is defensible on cost, latency and dependency count. If someone had written that tradeoff down I would have no post.

The point is that nobody chose it. It is a kwarg default on a helper class. There is no config surface for the judge model, no line in the README noting that the judge defaults to whatever the agent runs on, and — per that last grep — no mention anywhere in the repo of judge independence, model bias, or cross-model evaluation. The most consequential decision in an evaluation harness was never surfaced as a decision at all.

That is the failure mode I think is worth naming. Self-grading almost never arrives as a claim you can disagree with. It arrives as a default, three call frames down, in a constructor nobody reads.

And examples/ is precisely where defaults go to breed. Sample directories exist to be copied.

Where I am standing

I build an independent validation gate, so treat this as an interested party pointing at a thing that flatters him, and go run the command.

And the number that matters here is the unflattering one, so: on a 113-example internal holdout, our hosted API flagged 19.5% of the clean items — "clean" meaning an item our adjudicated labels recorded as containing no defect, so a flag on one is the gate being wrong, not the item. Banked 2026-08-10. That is our data on our holdout; yours will differ. A separate judge is not a correct judge. It is only a judge that is not marking its own work.

I also filed an unrelated bug in the same file — a debug exit() that stops the example running — as issue #4, with a fix in PR #5.