Grade the Grader First: A κ-Gate for Coding-Agent Benchmarks

You have forty patches. Two reviewers score them pass or fail against the same rubric. They agree on twenty-nine. So you compute the win rate, publish the ranking, and move on.

A week later someone reruns the sweep with a different reviewer and the ranking inverts. The agents did not change. The instrument did.

A patch label is a measurement, not a fact. If your two labelers cannot reproduce each other, no amount of endpoint comparison will save the result. This post is a workflow for checking that before you publish anything, and for refusing to publish when the check fails.

Why agreement comes before accuracy

Accuracy needs ground truth. For a coding patch you usually have something better and cheaper: a test command that either exits zero or does not. Start there.

But the interesting patches are the ones the tests do not settle. Refactors that keep tests green. Fixes that pass tests for the wrong reason. Partial fixes that satisfy two of three assertions. Those go to a human label, and that human label is where comparability dies quietly.

Step 1: Build a small, stratified, perturbed dataset

Forty tasks is enough for a calibration pass and small enough that two people can label it in an afternoon. Do not sample forty tasks from one difficulty band. Stratify deliberately.

bench/
  tasks/
    t-001/
      prompt.md          # the task as the agent sees it
      run.sh             # the check: exits 0 on pass
      mutate.py          # perturbation probe, optional
  labels/
    grader-a.csv
    grader-b.csv
  manifest.json

Three strata, roughly equal size:

1. Mechanical — lint, type error, import cycle. A correct patch is unambiguous.
2. Behavioral — one named test flips from red to green. Still mostly objective.
3. Judgment — refactor-only, or a fix where the test suite is a weak proxy.

If stratum 3 disagrees heavily and stratum 1 does not, you have learned something specific: your rubric is fine for mechanical work and unreliable for the work you actually care about. That is a useful finding, and it is invisible in a single blended score.

Add a perturbation probe while you are here. Rename the identifiers, shuffle the imports, reorder the tests. A patch that only passes on the original spelling of a function is a memorization signal, not a capability signal. It is a crude probe, but it costs ten minutes.

Step 2: Collect two independent labels

Independent means the second grader does not see the first grader's labels. Not the score, not the comments, not the diff annotations. Two humans reading the same patch cold.

Use a binary label plus an abstain option: pass, fail, unclear. Abstain is not a cop-out. It is a signal that the rubric is underspecified at that point, and it tells you exactly which clause to rewrite.

Keep a third labeler in reserve. Disagreements get adjudicated by that person and become your reportable subset. You publish the adjudicated subset plus the raw agreement, not one blended number.

Step 3: Compute κ before you compute anything else

Agreement that looks high can be pure chance. If 90% of patches pass, two graders who always say pass agree 90% of the time while measuring nothing. Cohen's κ corrects for that.

Reference implementation, no third-party dependencies:

# kappa.py
from collections import Counter

def cohen_kappa(a, b):
    assert len(a) == len(b) and a, 'need equal, non-empty label vectors'
    n = len(a)
    labels = sorted(set(a) | set(b))
    po = sum(1 for x, y in zip(a, b) if x == y) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum((ca[l] / n) * (cb[l] / n) for l in labels)
    if pe == 1.0:
        raise ValueError('both graders collapsed to one label; kappa undefined')
    return (po - pe) / (1 - pe), po

A point estimate on forty items is not enough. Bootstrap it:

import random

def bootstrap_kappa(a, b, iters=2000, seed=7):
    random.seed(seed)
    n = len(a)
    vals = []
    for _ in range(iters):
        idx = [random.randrange(n) for _ in range(n)]
        try:
            k, _ = cohen_kappa([a[i] for i in idx], [b[i] for i in idx])
        except (ValueError, AssertionError):
            continue
        vals.append(k)
    vals.sort()
    return vals[int(0.025 * len(vals))], vals[int(0.975 * len(vals))]

Then turn it into a gate, so the decision is mechanical and not a matter of mood on release day:

python kappa.py --grader-a labels/grader-a.csv \
                --grader-b labels/grader-b.csv \
                --gate 0.60
# exit 0 -> proceed to ranking
# exit 2 -> rubric revision required; no ranking published

Step 4: Read the result against a decision table

Thresholds are conventions, not physics. The bands below follow the well-known descriptive scale from Landis and Koch (Biometrics, 1977), which is widely cited for exactly this purpose. Cite it and move on; do not relitigate it in your results section.

κ (95% CI)	Reading	Action
< 0.40	Disagreement dominates	Do not publish a ranking. Rewrite the rubric to test-anchored binary criteria.
0.40 – 0.59	Marginal	Publish per-stratum agreement and the adjudicated subset only.
0.60 – 0.79	Substantial	Ranking is publishable with κ, CI, and n attached.
≥ 0.80	Strong	Publishable. Still report sample size and stratum breakdown.

Always report po (raw agreement) and the label marginals next to κ. A κ of 0.65 with 80% raw agreement means something different from a κ of 0.65 with 55% raw agreement.

Step 5: Run the calibration pass on a disposable box

This is where infrastructure matters more than people expect. A calibration sweep installs language toolchains, mutates repositories, and leaves test caches everywhere. You do not want any of that on your laptop, because next month you will want to rerun the same manifest and your local environment will have drifted.

A throwaway runner is the right shape: provision it, run the sweep, export labels/, manifest.json, and the harness logs, then destroy it. Same input, same box image, same output.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here for two operator-supplied availability claims that fit this workflow: free model access for the endpoint under test, and a free server option for the runner. The brief I was given also states a free allowance of 10M tokens, which is a reasonable size for a forty-task calibration pass — but free-tier terms and quotas change, so check the project page for the current numbers before you size a run and write those numbers into your manifest.

The reason this matters for methodology, not just cost: a benchmark that a stranger can rerun on free infrastructure with free endpoints is a benchmark that can be audited. Rankings that only reproduce on someone's paid cluster are closer to marketing than to measurement.

What this workflow does not fix

Be honest about the edges.

• κ measures agreement, not correctness. Two graders can agree confidently and both be wrong. Anchor to run.sh exit codes wherever possible, and treat human labels as the exception path.
• κ is prevalence-sensitive. Extreme pass rates inflate chance agreement and produce unstable κ values. Report marginals; if one label exceeds ~90% of the set, add harder tasks before trusting the number.
• Forty items gives a wide interval. That is fine for a gate. It is not fine for claiming a two-point difference between endpoints.
• Two graders only. For three or more, use Fleiss' κ or Krippendorff's α instead of averaging pairwise κ.
• The perturbation probe is crude. It catches blatant memorization, not subtle contamination. It is a smoke test, not a proof.

Who should skip this

If you already run continuous evaluation where every task has an executable check and labels are generated, not judged, you do not need a κ-gate — your instrument is already objective. If you cannot get two independent human labelers for two hours, this workflow will not run, and you should say so in your write-up rather than implying comparability you did not measure.

And if you only need a smoke test to see whether an agent can edit a file without breaking imports, skip all of this. A κ-gate is for the moment you are about to publish a ranking.

Run the gate first. If κ clears 0.60 with an interval you are willing to defend, the ranking that follows is worth reading. If it does not, you just saved yourself from publishing a number that would flip on the next run — and that is the cheapest result in the whole benchmark.