awslabs/Agent-EvalKit is a sample kit for evaluating AI agents. Its QA example defines the DeepEval judge in examples/qa_agent_evaluation/eval/metrics.py:
class BedrockLLM(DeepEvalBaseLLM):
"""Custom DeepEval LLM using LiteLLM to call Bedrock."""
def __init__(self, model: str = "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"):
Both metric factories instantiate it with no argument:
def create_faithfulness_metric(threshold: float = 0.7) -> FaithfulnessMetric:
bedrock_llm = BedrockLLM()
return FaithfulnessMetric(threshold=threshold, model=bedrock_llm, include_reason=True)
The agent under evaluation runs us.anthropic.claude-sonnet-4-20250514-v1:0 — that is gen_ai.request.model in the example's own eval/otel-traces.jsonl, and it is the only model id that appears anywhere in that trace file.
So the judge and the subject are the same model. (Not the same string: the judge carries LiteLLM's bedrock/ route prefix. Same model, one hop of indirection.) The bundled report gives the agent a faithfulness score of 78.2%, and the thing measuring faithfulness is the thing whose faithfulness is being measured.
Check it yourself:
curl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \
&& cd awslabs-Agent-EvalKit-*/examples/qa_agent_evaluation/eval \
&& echo "judge:" && grep -o 'bedrock/us\.anthropic\.[a-z0-9.:-]*' metrics.py | sort -u \
&& echo "subject:" && grep -o 'us\.anthropic\.[a-z0-9.:-]*' otel-traces.jsonl | sort -u \
&& echo "caveats:" && grep -rniE 'same model|self-grad|bias|independent|different model|cross-model|judge model' \
../../../ --include=*.md --include=*.py ; echo "^ nothing under 'caveats:' = undisclosed"
Confirmed at 472fbc6.
The part worth arguing about
Not "AWS shipped something bad." It is a Labs sample — 38 stars at the time of writing, last pushed in March 2026 — and using one Bedrock model for both roles is defensible on cost, latency and dependency count. If someone had written that tradeoff down I would have no post.
The point is that nobody chose it. It is a kwarg default on a helper class. There is no config surface for the judge model, no line in the README noting that the judge defaults to whatever the agent runs on, and — per that last grep — no mention anywhere in the repo of judge independence, model bias, or cross-model evaluation. The most consequential decision in an evaluation harness was never surfaced as a decision at all.
That is the failure mode I think is worth naming. Self-grading almost never arrives as a claim you can disagree with. It arrives as a default, three call frames down, in a constructor nobody reads.
And examples/ is precisely where defaults go to breed. Sample directories exist to be copied.
Where I am standing
I build an independent validation gate, so treat this as an interested party pointing at a thing that flatters him, and go run the command.
And the number that matters here is the unflattering one, so: on a 113-example internal holdout, our hosted API flagged 19.5% of the clean items — "clean" meaning an item our adjudicated labels recorded as containing no defect, so a flag on one is the gate being wrong, not the item. Banked 2026-08-10. That is our data on our holdout; yours will differ. A separate judge is not a correct judge. It is only a judge that is not marking its own work.
I also filed an unrelated bug in the same file — a debug exit() that stops the example running — as issue #4, with a fix in PR #5.
Top comments (2)
Couldn't unsee this once I saw it in our setup. Our chain-of-thought quality judge turned out to be fine-tuned on the same base model family as the agent, and it took three months and a provider switch to even notice the scores looked wrong. We substituted a judge from a different family and the pass rate dropped 23 points on the same holdout set. Still working out how much of the gap was defects the original judge was too calibrated to flag versus the new one being noisier on a different axis.
The 23 points is the difference between two judges — it isn't a measurement of either one. Neither judge can tell you how much of it is real, because each is the thing being questioned. You need a third reference that isn't a model.
The cheapest one that works: take 80–100 items from the same holdout, have two people label them pass/fail independently (keep the disagreements — they're informative), then run both judges on that set. Now every item lands in one of four cells, and the two cells you care about are:
Count those two cells and you've partitioned the 23 points. Everything else is agreement and doesn't move the answer.
Two things that make the number trustworthy rather than just plausible: run each judge three times over the same items and treat its disagreement with itself as the noise floor before you attribute anything to the other judge; and hold out the labelled set from any prompt tuning you do afterwards, or the next judge will be calibrated to your labels the same way the first was calibrated to the agent.
The pattern you describe — same base family, scores look fine for months, everything moves when you swap — is the same thing as the post, just inherited instead of hand-written. Same-family judges share blind spots with the model they grade, so their false-negative rate on exactly the failure modes that matter is the part you can't see from inside.
If it helps as a worked reference, AWS's
sample-gen-ai-evaluations-workshopships a notebook that does precisely this evaluate-the-judge step (02-quality-metrics/03_Evaluating_your_Judge.ipynb: human-labelled benchmark, held-out split, TPR/TNR per judge, repeatability, and a bias check for verbosity and confidence). It's a good template regardless of stack.