DEV Community

michael hurst
michael hurst

Posted on

An AWS Labs agent-eval sample uses the same model as judge and subject

#ai #llm #testing #opensource

awslabs/Agent-EvalKit is a sample kit for evaluating AI agents. Its QA example defines the DeepEval judge in examples/qa_agent_evaluation/eval/metrics.py:

class BedrockLLM(DeepEvalBaseLLM):
    """Custom DeepEval LLM using LiteLLM to call Bedrock."""

    def __init__(self, model: str = "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"):
Enter fullscreen mode Exit fullscreen mode

Both metric factories instantiate it with no argument:

def create_faithfulness_metric(threshold: float = 0.7) -> FaithfulnessMetric:
    bedrock_llm = BedrockLLM()
    return FaithfulnessMetric(threshold=threshold, model=bedrock_llm, include_reason=True)
Enter fullscreen mode Exit fullscreen mode

The agent under evaluation runs us.anthropic.claude-sonnet-4-20250514-v1:0 — that is gen_ai.request.model in the example's own eval/otel-traces.jsonl, and it is the only model id that appears anywhere in that trace file.

So the judge and the subject are the same model. (Not the same string: the judge carries LiteLLM's bedrock/ route prefix. Same model, one hop of indirection.) The bundled report gives the agent a faithfulness score of 78.2%, and the thing measuring faithfulness is the thing whose faithfulness is being measured.

Check it yourself:

curl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \
  && cd awslabs-Agent-EvalKit-*/examples/qa_agent_evaluation/eval \
  && echo "judge:"   && grep -o 'bedrock/us\.anthropic\.[a-z0-9.:-]*' metrics.py | sort -u \
  && echo "subject:" && grep -o 'us\.anthropic\.[a-z0-9.:-]*' otel-traces.jsonl | sort -u \
  && echo "caveats:" && grep -rniE 'same model|self-grad|bias|independent|different model|cross-model|judge model' \
       ../../../ --include=*.md --include=*.py ; echo "^ nothing under 'caveats:' = undisclosed"
Enter fullscreen mode Exit fullscreen mode

Confirmed at 472fbc6.

The part worth arguing about

Not "AWS shipped something bad." It is a Labs sample — 38 stars at the time of writing, last pushed in March 2026 — and using one Bedrock model for both roles is defensible on cost, latency and dependency count. If someone had written that tradeoff down I would have no post.

The point is that nobody chose it. It is a kwarg default on a helper class. There is no config surface for the judge model, no line in the README noting that the judge defaults to whatever the agent runs on, and — per that last grep — no mention anywhere in the repo of judge independence, model bias, or cross-model evaluation. The most consequential decision in an evaluation harness was never surfaced as a decision at all.

That is the failure mode I think is worth naming. Self-grading almost never arrives as a claim you can disagree with. It arrives as a default, three call frames down, in a constructor nobody reads.

And examples/ is precisely where defaults go to breed. Sample directories exist to be copied.

Where I am standing

I build an independent validation gate, so treat this as an interested party pointing at a thing that flatters him, and go run the command.

And the number that matters here is the unflattering one, so: on a 113-example internal holdout, our hosted API flagged 19.5% of the clean items — "clean" meaning an item our adjudicated labels recorded as containing no defect, so a flag on one is the gate being wrong, not the item. Banked 2026-08-10. That is our data on our holdout; yours will differ. A separate judge is not a correct judge. It is only a judge that is not marking its own work.

I also filed an unrelated bug in the same file — a debug exit() that stops the example running — as issue #4, with a fix in PR #5.

Top comments (0)