Couldn't unsee this once I saw it in our setup. Our chain-of-thought quality judge turned out to be fine-tuned on the same base model family as the agent, and it took three months and a provider switch to even notice the scores looked wrong. We substituted a judge from a different family and the pass rate dropped 23 points on the same holdout set. Still working out how much of the gap was defects the original judge was too calibrated to flag versus the new one being noisier on a different axis.
The 23 points is the difference between two judges — it isn't a measurement of either one. Neither judge can tell you how much of it is real, because each is the thing being questioned. You need a third reference that isn't a model.
The cheapest one that works: take 80–100 items from the same holdout, have two people label them pass/fail independently (keep the disagreements — they're informative), then run both judges on that set. Now every item lands in one of four cells, and the two cells you care about are:
old judge PASS, new judge FAIL, humans FAIL → defects the original judge was too calibrated to flag
old judge PASS, new judge FAIL, humans PASS → the new judge being noisier
Count those two cells and you've partitioned the 23 points. Everything else is agreement and doesn't move the answer.
Two things that make the number trustworthy rather than just plausible: run each judge three times over the same items and treat its disagreement with itself as the noise floor before you attribute anything to the other judge; and hold out the labelled set from any prompt tuning you do afterwards, or the next judge will be calibrated to your labels the same way the first was calibrated to the agent.
The pattern you describe — same base family, scores look fine for months, everything moves when you swap — is the same thing as the post, just inherited instead of hand-written. Same-family judges share blind spots with the model they grade, so their false-negative rate on exactly the failure modes that matter is the part you can't see from inside.
If it helps as a worked reference, AWS's sample-gen-ai-evaluations-workshop ships a notebook that does precisely this evaluate-the-judge step (02-quality-metrics/03_Evaluating_your_Judge.ipynb: human-labelled benchmark, held-out split, TPR/TNR per judge, repeatability, and a bias check for verbosity and confidence). It's a good template regardless of stack.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Couldn't unsee this once I saw it in our setup. Our chain-of-thought quality judge turned out to be fine-tuned on the same base model family as the agent, and it took three months and a provider switch to even notice the scores looked wrong. We substituted a judge from a different family and the pass rate dropped 23 points on the same holdout set. Still working out how much of the gap was defects the original judge was too calibrated to flag versus the new one being noisier on a different axis.
The 23 points is the difference between two judges — it isn't a measurement of either one. Neither judge can tell you how much of it is real, because each is the thing being questioned. You need a third reference that isn't a model.
The cheapest one that works: take 80–100 items from the same holdout, have two people label them pass/fail independently (keep the disagreements — they're informative), then run both judges on that set. Now every item lands in one of four cells, and the two cells you care about are:
Count those two cells and you've partitioned the 23 points. Everything else is agreement and doesn't move the answer.
Two things that make the number trustworthy rather than just plausible: run each judge three times over the same items and treat its disagreement with itself as the noise floor before you attribute anything to the other judge; and hold out the labelled set from any prompt tuning you do afterwards, or the next judge will be calibrated to your labels the same way the first was calibrated to the agent.
The pattern you describe — same base family, scores look fine for months, everything moves when you swap — is the same thing as the post, just inherited instead of hand-written. Same-family judges share blind spots with the model they grade, so their false-negative rate on exactly the failure modes that matter is the part you can't see from inside.
If it helps as a worked reference, AWS's
sample-gen-ai-evaluations-workshopships a notebook that does precisely this evaluate-the-judge step (02-quality-metrics/03_Evaluating_your_Judge.ipynb: human-labelled benchmark, held-out split, TPR/TNR per judge, repeatability, and a bias check for verbosity and confidence). It's a good template regardless of stack.