Calibrating an LLM-as-Judge for a Vision Check to κ 0.80

At AutoKon, a construction platform where I was founding CTO and sole engineer, a contractor's payment was released only after the field photos passed an AI check. Claude Opus Vision, with Pinecone retrieval for context, described what each photo showed. A separate LLM-as-judge pipeline in Arize AX graded those descriptions for groundedness and completeness, and the judges had to reach an agreement score of κ ≥ 0.80 against human review before their results counted toward payment. The evals lived in a version-controlled repo and ran in CI with phoenix-evals and GitHub Actions.

A note on scope before the method. I am not sharing dataset sizes, prompt text, label counts or per-round scores here. Where I describe a step below, treat it as the general method I use and recommend, written so you can apply it to your own vision task. I am not reproducing the exact AutoKon configuration.

Why a judge needs calibrating at all

The generator model writes descriptions. A judge model grades them, which lets you score every output instead of a hand-picked sample. The catch is obvious once you say it out loud: the judge is also a model, and you have no idea how good it is until you compare it with people.

Calibration means measuring how often the judge agrees with careful human reviewers, adjusting it until that agreement is high enough, and then holding it there. Without that step, a dashboard full of judge scores is a number nobody has checked.

Define the rubric before you write the judge prompt

Most of the work is here. If two humans cannot agree on what "good" means, no judge will either.

For photo descriptions, I split quality into two separate dimensions:

• Groundedness. Is every claim in the description visible in the photo? A description that says "plastering complete on the east wall" when the east wall is out of frame fails, even if the claim happens to be true.
• Completeness. Does the description cover the items that matter for this stage of work? A grounded but vague description ("a wall is visible") can still be useless for a payment decision.

Rubric habits that help agreement:

1. Prefer binary labels at first. Pass or fail is easier for humans to agree on than a 1 to 5 scale. Move to an ordinal scale only if binary loses information you actually need.
2. One judge prompt per dimension. Asking one prompt to score both lets a strong score on one bleed into the other.
3. Write each criterion as a question that can be answered from the image and description alone. If the reviewer needs outside knowledge, the judge needs it too, so put it in the prompt or drop the criterion.
4. Include worked examples of pass and fail, especially borderline ones. Borderline cases are where raters disagree.
5. Give the judge the image. A text-only judge grading a vision model's output can only check whether the text sounds plausible. A vision judge can check the text against the pixels.

Build a human-labeled set

The general method:

• Sample for coverage. Pull photos across lighting conditions, angles, stages of work, blurry shots, and cases where the generator is known to struggle.
• Oversample failures. If 95% of descriptions are fine, a random sample gives you very few failures to calibrate against, and kappa becomes unstable.
• Have two people label independently using the written rubric, without seeing each other's labels.
• Measure human to human agreement first. That number is your practical ceiling. If two humans only reach 0.70, a judge reaching 0.80 against either one is more likely a fluke than a triumph, and the rubric needs work.
• Adjudicate disagreements into a single gold label, and update the rubric wording whenever a disagreement reveals ambiguity.
• Split the set into a calibration portion you iterate against and a held-out portion you touch only to confirm.

Why kappa and not accuracy

Accuracy rewards a judge for agreeing on the easy majority class. Cohen's kappa corrects for the agreement you would expect by chance:

κ = (p_o - p_e) / (1 - p_e)

p_o is observed agreement and p_e is the agreement expected if both raters labeled at random with their own label frequencies. For two raters and categorical labels, Cohen's kappa is the standard choice. For ordinal scales, weighted kappa gives partial credit for near misses.

On the common Landis and Koch scale, 0.61 to 0.80 counts as substantial agreement and 0.81 to 1.00 as almost perfect, so a 0.80 bar sits right at that boundary.

Here is a small, dependency-free implementation that shows why accuracy misleads:

def cohens_kappa(rater_a, rater_b):
    """Cohen's kappa for two raters labeling the same items."""
    if len(rater_a) != len(rater_b) or not rater_a:
        raise ValueError("need two equal-length, non-empty label lists")
    n = len(rater_a)
    labels = set(rater_a) | set(rater_b)
    p_o = sum(a == b for a, b in zip(rater_a, rater_b)) / n
    p_e = sum((rater_a.count(k) / n) * (rater_b.count(k) / n) for k in labels)
    if p_e == 1:
        return float("nan")  # both raters used one identical label: undefined
    return (p_o - p_e) / (1 - p_e)


human = ["pass"] * 18 + ["fail"] * 2
lazy_judge = ["pass"] * 20
better_judge = ["pass"] * 17 + ["fail"] + ["fail"] * 2

for name, judge in [("lazy", lazy_judge), ("better", better_judge)]:
    accuracy = sum(h == j for h, j in zip(human, judge)) / len(human)
    print(f"{name}: accuracy={accuracy:.2f} kappa={cohens_kappa(human, judge):.2f}")

Output:

lazy: accuracy=0.90 kappa=0.00
better: accuracy=0.95 kappa=0.77

The lazy judge passes everything and still scores 90% accuracy, with a kappa of zero. The better judge catches both real failures and makes one false rejection. It reaches 95% accuracy and still falls short of 0.80. That gap between a flattering accuracy number and an honest kappa is the whole reason to use kappa on imbalanced data. (These are toy labels for illustration, not AutoKon data.)

The calibration loop

1. Run the judge on the calibration split.
2. Compute kappa per dimension, separately. A combined score hides which dimension is weak.
3. Read every disagreement. Build a confusion matrix and look at the actual photos. Is the judge too strict on blurry images? Too lenient when the description is long?
4. Change one thing at a time: rubric wording, examples, the instruction about uncertain cases, or the judge model.
5. Re-run and re-score.
6. Once the calibration split clears 0.80, confirm on the held-out split. If held-out drops noticeably, you tuned the prompt to the examples you were staring at.
7. Freeze the judge: pin the prompt, model version and rubric in version control, and have CI re-run the agreement check whenever any of them change.

Pitfalls

• Overfitting the prompt to the calibration set. The held-out split exists for this.
• Shared blind spots. If the judge sees exactly the same context as the generator, they can fail in the same way. Giving the judge the raw image, and asking it to check claims against it, reduces this.
• Verbosity bias. Judges tend to rate longer descriptions as more complete. Include short complete examples and long incomplete ones in your few-shot set.
• Silent drift. A model version change or a small prompt edit can move agreement. Treat the judge like code.
• Low failure prevalence. With very few failures, kappa swings a lot between runs. Oversample failures and report the case counts behind each score.
• Label drift in humans. Reviewers' standards shift over time. Re-measure human to human agreement when you refresh labels.
• Treating kappa as the only signal. Kappa measures agreement with people, so keep a domain expert involved in what the rubric asks.

Where the judge sits

At AutoKon the AI check was one input to a five-role approval flow (field worker, site manager, project manager, finance, owner), with two rounds of photo evidence, Opname and BASTP, before payment. A calibrated judge made the AI's descriptions measurable. Accountable people still approved the payment, and I would design any money-touching system the same way.

If you are building evals for a vision or LLM feature and want to compare notes, you can find me at https://faishalwahiduddin.com/?utm_source=devto&utm_medium=article&utm_campaign=blog-network-2026-10

Comments

[Dean Lee]

Calibrating an LLM judge against human agreement to kappa 0.80 is the right discipline for turning vision evals into an enforceable payment gate. Without empirical inter-rater agreement, automated verification is just an uncalibrated point estimate.

The operational catch in milestone payment releases is payoff asymmetry. Releasing funds on an incomplete installation creates immediate capital loss that is painful to claw back from a contractor. Withholding funds on completed work introduces contractor dispute friction, but the cash remains safely inside escrow.

If a judge is optimized for symmetric agreement like unweighted kappa rather than an asymmetric loss function, a high score can still leak capital on borderline passes. The real economic leverage in an evaluation pipeline comes from routing the distribution. Let the calibrated judge clear the unambiguous interior automatically, and shunt the borderline density into manual review where the marginal cost of human inspection is dwarfed by the downside tail risk of an invalid disbursement.

[Arhan Canli]

I ran your toy example and the numbers hold: accuracy 0.95 and kappa 0.77 for the better judge, 0.90 and 0.00 for the lazy one.

One thing the example shows without saying it: with 2 failures in 20 items, a single disagreement decides whether you clear the bar. Keep the same 18/2 split and let the judge make one false rejection and kappa is 0.77. Let it miss one real failure instead and it is 0.64. Both are one item. Scale the set to 200 items with 20 failures (same 10% rate) and one disagreement of either kind gives 0.97, so the 0.80 gate means something there and almost nothing at 20.

So I'd report the failure count next to every kappa, and add an interval (bootstrap over items) before a result is allowed to count toward payment. On a gate where money is released, a lower bound above 0.80 is a stricter and more honest test than a point estimate above it. Do you resample items or documents when you bootstrap, since several photos of the same job are probably correlated?