I spent a week last month watching an agent "improve" itself and score worse on every single eval run. Not because the agent was getting worse — because the judge was.
The paper makes the point cleanly: user feedback carries a signal that LLM judges systematically can't detect. When a model revises an answer based on real user feedback, humans rate the revision as better. An LLM-as-judge rates the same revision as worse. Not neutral — actively biased against the improvement.
If you're running any kind of agent loop that evaluates its own outputs, that should stop you cold. It means your eval isn't just noisy. It's selecting for the wrong thing.
Why the judge is blind
Think about what a feedback-informed revision actually looks like. The model was told "this is wrong because X," so it changes the answer. That change usually involves hedging, acknowledging a mistake, narrowing a claim, or adding a caveat. All of those are exactly the linguistic patterns an LLM judge has been trained to treat as lower quality.
Judges reward confidence. They reward clean, assertive prose. A revision that says "I previously said Y, but on reflection Z is more accurate" reads as less confident — so it scores lower. The model did the right thing, and the judge punishes it for the honesty.
That's the trap. The signal that makes an output genuinely better — responsiveness to feedback — is the signal the judge is structurally blind to. The judge was trained on static text quality, not on whether the text got better.
I've seen this play out in a concrete way. We had an agent that would revise a plan after a user pointed out a missing constraint. The revised plan was tighter, more realistic, actually deployable. The judge consistently ranked the original — the one missing the constraint — higher. The original was more confident. It was also wrong. The judge couldn't tell the difference, because it was scoring prose, not correctness.
What this does to agent loops
The whole premise of a self-improving agent is: do the thing, evaluate, keep what's good, discard what's not. If your evaluator can't recognize improvement, the loop converges on the wrong optimum. You're not just losing a few points — you're actively discarding your best outputs and keeping the mediocre ones.
I've seen this in the wild more than I'd like. Teams ship an eval harness, the agent iterates, the scores go sideways, and the conclusion is "the model can't improve." Nine times out of ten the model was fine. The judge was the problem.
The worst part is how quiet it is. A bad judge doesn't fail loudly. It produces plausible scores that drift in the wrong direction, and you spend a week debugging the agent before you think to question the eval. The eval is the last thing anyone suspects, because it's the thing you built to be objective.
What I'd try instead
I don't have a clean fix, and I'd be suspicious of anyone who says they do. But I've got a few things that have helped.
First, stop treating LLM-as-judge as ground truth for anything involving revision. Use it for ranking static outputs if you must, but the moment feedback enters the picture, the judge's opinion is suspect.
Second, prefer behavioral signals over vibes. Did the agent actually fix the thing the feedback pointed at? That's checkable. Parse the feedback, extract the claim, verify the revision addresses it. That's a deterministic check, not a judgment call. It's more work to build, but it doesn't have the bias baked in.
Third, keep a human in the loop for the small set of outputs that actually matter. I know, it's not scalable. But a hundred human spot-checks a week will catch a biased judge faster than a million automated evals will.
And fourth — log everything. When your scores drift, you need to be able to ask "did the judge change, or did the output change?" If you can't answer that, you're flying blind.
The uncomfortable part
Maybe I'm wrong here, but I think this cuts deeper than eval hygiene. If LLM judges can't detect the value of feedback, then any system that uses them to close the loop is structurally limited. The model can only improve as far as its judge can see. Blind judge, blind ceiling.
That's a real constraint on the whole "agents that learn from their mistakes" pitch. Not because the models can't learn — but because we've built the evaluation layer on a tool that can't perceive the thing we're trying to optimize for.
The paper's worth reading for that reason alone. It's not a critique of one judge. It's a critique of the assumption that an LLM can grade what another LLM learned. https://arxiv.org/abs/2609.02859v1
I'm going to go re-check my eval harness now. I suspect it's been lying to me.
Top comments (0)