Sixteen out of sixteen. Every summary my eval graded came back faithful, first try. That number should make you suspicious of the eval, not proud of the system, and it's the reason this post exists: a same-day Bedrock experiment on my own blog, built to find out what it actually takes to trust an LLM-as-judge score before you wire one into anything that matters.
The setup
Sixteen posts, two Bedrock models in my own AWS account, one small repo: do-you-trust-it-evals.
-
Claude Haiku 4.5 (
us.anthropic.claude-haiku-4-5-20251001-v1:0) writes a 2-3 sentence summary of each of my 16 published posts, instructed to use only claims present in the source. -
Claude Sonnet 4.5 (
us.anthropic.claude-sonnet-4-5-20250929-v1:0) grades each summary against its source: faithfulness 1-5, completeness 1-5, a list of unsupported claims, and a PASS/FAIL verdict. The rubric treats a conditional claim stated as a universal ("sometimes" becoming "always") as a hallucination, not just an invented fact.
Round 1: the baseline that should worry you
Cold run, no tuning: 16 out of 16 summaries passed, faithfulness 5 across the board. I didn't take the judge's word for it. The summary for the CDK post claims "nine specific pitfalls", and the source has exactly nine, numbered ## Gotcha #1 through ## Gotcha #9. The summaries were genuinely faithful, not a rubber stamp catching nothing because there was nothing to catch.
A 100% pass rate proves the eval didn't break on the easy case. It proves nothing about whether the judge would catch a hard one.
Round 2: calibrate with a single variable
Five corrupted summaries, each with exactly one injected error and nothing else touched, so a FAIL verdict can only be explained by that one change: a fabricated number, a conditional claim broadened to a universal, a real number misattributed to the wrong post, a fabricated named entity, an inflated count.
5 out of 5 caught, and in every case the judge's own hallucination list named the exact injected error. If you can't point to the one thing you changed in a corrupted test case, you don't have a calibration result, you have a guess with a percentage attached.
Round 3: does the judge agree with itself
Fiona Lau's "Same Input, Different Scores" (2026) found substantial LLM-judge score variability even at temperature 0, with completeness scoring showing the largest fluctuations. So I re-ran two inputs, an easy clean one and a hard borderline one, four times each, tracking both faithfulness and completeness. Both held steady across all eight runs.
What I now do
- Treat a 100% pass rate as an instruction to calibrate, not a result to report.
- Corrupt one variable per test case, and diff it against the original to check.
- Cover more than one failure category: a fabricated fact, a scope-broadened claim, and a misattributed-but-real number all fail differently.
- Re-grade the same unmodified input more than once before trusting a single run.
Full methodology, all three scripts, and the raw JSON for every round: github.com/rajmurugan01/do-you-trust-it-evals.
This is Part 1 of Do You Trust It?, a series on how you actually know an AI system is good once it's live. Full write-up with the honest caveats (small n, benign-only corruptions, what this doesn't test) on rajmurugan.com.
Top comments (0)