How to Audit Bias When Free LLMs Judge Paid Models

Free models can judge paid models at near-zero cost, but only after you audit the judge itself. Predictable bias can be calibrated; unaudited bias will silently corrupt every ranking in the pipeline.

Evaluation pipelines burn judgments. Each judgment is an API call. Paid judges get expensive at scale; free judges cost near zero. MonkeyCode's free tier — 10 million tokens and a free server option — makes large-scale judging feasible. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I do not ask whether a free judge is perfect. I ask whether its bias is predictable. Predictable bias is a calibration problem. Unpredictable bias is fatal.

Design a Blind Free-vs-Paid Judge Audit

I use one protocol: three judges, two hundred samples, five criteria, one script.

• One free model rates all two hundred samples.
• One paid model rates all two hundred samples.
• Humans rate a fifty-sample subset as the calibration slice.
• Samples come from real work: summarization, extraction, and code explanation.
• Each sample is a task prompt plus a model response. Judges never see which model wrote the response.

That last rule is the audit. If the judge knows the source model, self-preference documented in LLM-as-a-judge research contaminates the score. Keep source identity out of the payload.

Two hundred is not magic. It is large enough to split by task type and still estimate chance-corrected agreement. Fifty human ratings are a calibration set, not a full gold standard. Compare this to a single 1–10 overall score: one number hides which axis drifted. Five axes—accuracy, clarity, completeness, conciseness, tone—make the drift visible.

Blind, multi-criterion, mixed-task: that is the design. Anything looser is a vibe check, not an audit.

Lock the Judge Prompt and Run One Script

Prompt design determines bias more than model size. I pin temperature at zero, force JSON, and define every criterion in the system message. Zero temperature reduces sampling noise so disagreement is bias, not luck. JSON-only output makes parsing deterministic. Five criteria give fine-grained signals; one criterion hides bias.

The judge prompt

JUDGE_PROMPT = """You are an expert evaluator. Rate the response on five criteria.
Use ONLY integers 1-5.
Return JSON only, no prose.

Criteria:
- accuracy: factual correctness
- clarity: readability and structure
- completeness: coverage of the request
- conciseness: information density
- tone: professional and neutral

{"accuracy": int, "clarity": int, "completeness": int, "conciseness": int, "tone": int}"""

Keep the schema in the prompt. Models that “almost” follow instructions will still emit prose; the parser must fail closed, not invent a score.

The audit script

One script hits both OpenAI-compatible endpoints and writes a single JSON file. Two HTTP clients, one results file, no hidden state.

# judge_audit.py — compare free vs paid LLM judges
import asyncio
import json
import os

import httpx

FREE_BASE = os.environ["FREE_BASE_URL"]
FREE_MODEL = os.environ["FREE_MODEL"]
PAID_BASE = os.environ["PAID_BASE_URL"]
PAID_MODEL = os.environ["PAID_MODEL"]
API_KEY = os.environ["API_KEY"]

JUDGE_PROMPT = """You are an expert evaluator. Rate the response on five criteria.
Use ONLY integers 1-5. Return JSON only, no prose.
Criteria: accuracy, clarity, completeness, conciseness, tone.
{"accuracy": int, "clarity": int, "completeness": int, "conciseness": int, "tone": int}"""


def build_payload(model, task, response):
    return {
        "model": model,
        "messages": [
            {"role": "system", "content": JUDGE_PROMPT},
            {"role": "user", "content": f"Task: {task}\n\nResponse: {response}"},
        ],
        "temperature": 0.0,
        "max_tokens": 150,
    }


async def judge_one(client, base, model, task, response):
    r = await client.post(
        f"{base}/chat/completions",
        json=build_payload(model, task, response),
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=60,
    )
    r.raise_for_status()
    text = r.json()["choices"][0]["message"]["content"]
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        return {"error": text[:200]}


async def main():
    samples = json.load(open("samples.json"))
    results = []
    async with httpx.AsyncClient() as client:
        for s in samples:
            free = await judge_one(client, FREE_BASE, FREE_MODEL, s["task"], s["response"])
            paid = await judge_one(client, PAID_BASE, PAID_MODEL, s["task"], s["response"])
            results.append({
                "id": s["id"],
                "free": free,
                "paid": paid,
                "human": s.get("human"),
            })
    json.dump(results, open("judge_results.json", "w"), indent=2)
    print("Wrote judge_results.json")


if __name__ == "__main__":
    asyncio.run(main())

Run it:

export FREE_BASE_URL="https://your-free-endpoint.example/v1"
export FREE_MODEL="free-model"
export PAID_BASE_URL="https://your-paid-endpoint.example/v1"
export PAID_MODEL="paid-model"
export API_KEY="your-key"
python judge_audit.py

This script is an audit tool, not a production judge. If JSON parse fails, I store the raw snippet under error instead of coercing a number. Fake scores are worse than missing scores.

Measure Agreement, Then Hunt Known Biases

Agreement is the reliability signal. I use two metrics: Cohen's kappa for categorical agreement on the 1–5 scores, and Spearman correlation for ranking agreement.

def cohen_kappa(r1, r2):
    n = len(r1)
    observed = sum(a == b for a, b in zip(r1, r2)) / n
    from collections import Counter
    c1, c2 = Counter(r1), Counter(r2)
    expected = sum((c1[v] / n) * (c2[v] / n) for v in set(c1) | set(c2))
    return (observed - expected) / (1 - expected) if expected < 1 else 1.0

How I read the numbers:

• Kappa above 0.6: substantial agreement — candidate for calibrated screening.
• Kappa below 0.4: weak agreement — do not trust that criterion from the free judge.
• Free-versus-human kappa is the critical measurement. Paid-versus-human kappa is the baseline.

Compute kappa per criterion, not on a blended average. A free judge that matches humans on accuracy but collapses on tone is usable for extraction and unusable for style. Blending those two stories into one number is how bad judges sneak into production.

Three bias patterns to compute, not guess

The literature already names the traps. Zheng et al. on LLM-as-a-judge with MT-Bench and Wang et al. on LLMs as unfair evaluators document self-preference, verbosity, and position effects. I treat them as tests, not essays.

1. Self-preference bias. A model inflates scores for its own outputs and deflates competitors. Detection: split results into self-judgments versus cross-judgments and compare mean scores per criterion.
2. Verbosity bias. Longer outputs get higher scores regardless of quality. Detection: correlate each criterion with response length. High correlation is the bias, not a quality signal.
3. Position bias. The first listed criterion absorbs extra weight. Detection: swap criterion order in the prompt, re-run a subset, and compare.

Free models can amplify these patterns: narrower training data, weaker instruction following. The audit is how I find out whether that amplification is small enough to calibrate. Guessing from a handful of cherry-picked examples is not an audit.

Decide When a Free Judge Is Trustworthy

Task type, output length, and criterion count decide trust—not the model card.

Structured tasks first. Extraction, classification, and formatting produce stable scores. Subjective tasks—creative writing, style, tone—are where free judges fail in my runs.

Prefer short outputs. Responses under two hundred words score more stably. Long outputs expose verbosity and position bias because there is more text for the judge to overweight.

Never collapse to one criterion. Five criteria beat one. A single score hides which bias fired. If completeness and conciseness move in opposite directions, that disagreement is the finding.

Limitations I will not paper over:

• The script is an audit tool, not a production judge. Production use needs calibration data, and calibration data comes from this audit.
• Free-tier quotas change. Verify current terms. Free models change. Re-run monthly.
• Do not use this for high-precision evaluation, compliance audits, or legal documents. Those need human judges. A free model is a screening tool, not a final arbiter.

Free models can judge paid models. Bias is often predictable. Predictable bias can be calibrated. A calibrated free judge is a screening layer, not a court of appeal.

Run the audit this week. Point the script at your free and paid endpoints, export judge_results.json, compute kappa against your human subset, and plot score versus length before you trust a single ranking. If a criterion surprises you, drop it from the free judge and keep it on the paid or human path. Measure, then decide—do not guess.