Originally published on AI Tech Connect.
What you need to know The judge is a system you build, not a prompt you paste. An LLM-as-a-judge that you have not designed and calibrated is just a second, unvalidated model whose opinions you happen to trust. Decide pointwise or pairwise first. Absolute scores gate releases; head-to-head comparisons rank changes. The choice shapes everything downstream, including which biases you have to fight. Write it as a rubric with explicit steps. The G-Eval pattern — chain-of-thought plus a form-filling schema — turns "rate the quality" into a repeatable procedure that two runs will agree on. Three biases are well documented. Position bias, verbosity or length bias and self-preference bias all have named mitigations: randomise order, length-normalise, and judge with a different model family.…
Top comments (0)