Score the Patch, Not the Story: A Coding-Agent Evaluation Schema

A teammate drops a number into Slack on a Friday: the agent hit 81% on “our coding suite.” You ask where the tasks came from. Silence. You ask whether the agent could read the unit tests. More silence. Two timeouts were deleted so the average would look cleaner. That 81% is a story. It is not a measurement.

You do not need a giant public leaderboard to stop this. You need a dataset schema, a handful of metrics that refuse to collapse into one float, and a rule for when a number may leave the lab. This article is a protocol you can run on a small box. Treat every code block as a working example, not a published result from a particular model.

What you are actually scoring

A coding agent is not a chatbot with a repo mounted. It is a patch factory. The only artifact that should enter a score is the diff, plus whether hidden tests and hidden constraints still hold after you apply that diff.

Visible tests are contaminated the moment they sit in the prompt, the tool output, or a CI log the agent can cat. If you average them with hidden tests, you are scoring reading comprehension of your own suite. Stop doing that.

Public GitHub issues are a second contamination path. If the task text, the failing test names, or the gold patch exist on the open web, a pretrained model may be recalling, not repairing. You still can use those tasks. You just cannot mix them with internal tickets and call the blend a single pass rate.

The four fields the dataset must carry

Every task file needs identity, isolation, provenance, and a locality bound. Miss one, and you are back to storytelling.

# tasks/PAY-1847.yaml  — example schema, not a live benchmark row
id: PAY-1847
origin: internal_ticket          # internal_ticket | synthetic | public_web
leakage_risk: low                # low | unknown | high
prompt: |
  Refunds posted after 17:00 UTC are dated the next business day.
  The batch job currently stamps local time. Fix the stamp, do not
  rewrite the ledger format.
allowed_files:
  - src/billing/batch_stamp.py
  - src/billing/business_day.py
max_files_touched: 2
max_diff_loc: 80
timeout_sec: 180
visible_tests: []                # empty on purpose
hidden_tests:
  - tests/test_batch_stamp_utc.py
hidden_constraints:
  - "Do not change CSV column order in ledger export"
  - "Stamp timezone must be UTC, not the server locale"

origin and leakage_risk are not metadata for a blog post. They are strata. You score each stratum alone. allowed_files and max_files_touched stop an agent from “winning” by rewriting the test harness. hidden_constraints catch the patch that goes green and still ships the wrong business rule.

If a task cannot fill those fields, it does not enter the set. You will have fewer tasks. That is the point.

Build the set in six numbered steps

1. Start from closed work, not from Twitter. Export 20–40 tickets your team already shipped. Prefer bugs with a failing test that landed in the same pull request as the fix. Synthetic tasks are allowed when you write both the bug and the hidden test yourself.
2. Strip the gold patch from disk. The runner image should contain the broken tree, the prompt, and the hidden tests. It should not contain expected.diff. If the agent can open the answer, you are measuring cat.
3. Split tests before the first run. Visible tests may teach. Hidden tests may only judge. Put hidden tests in a path the tools cannot list, or mount them after the agent exits. Label the mount in the run record.
4. Tag provenance on the same day you copy the text. A public issue cloned “just for coverage” is origin: public_web and leakage_risk: high even if you reword the title. Do not launder it into internal_ticket.
5. Freeze IDs. Once PAY-1847 is in the set, you do not edit its prompt because an agent failed it. You open PAY-1847.v2 as a new row. Otherwise your history is a moving target.
6. Write an exclusion log before you compute anything. Timeouts, checkout failures, missing interpreters, and network denials are not agent misses. They are rows in exclusions.jsonl. A protocol that drops them quietly is an ad.

Do the six steps on a sample of ten tasks before you scale. If you cannot fill the YAML, the suite is not ready. The agent is irrelevant until the suite is ready.

Metrics that refuse to become a headline

You report a row, not a percentage. Each completed run produces the object below. Average nothing across strata. Average nothing across unequal timeouts.

# score_patch.py — example scorer, unexecuted on any vendor model
from dataclasses import dataclass, asdict
from pathlib import Path
import json, subprocess, textwrap

@dataclass
class TaskScore:
    task_id: str
    origin: str
    leakage_risk: str
    hidden_pass: bool
    constraint_hits: int
    constraint_total: int
    files_touched: int
    files_allowed: int
    excess_files: int
    diff_loc: int
    max_diff_loc: int
    locality_ok: bool
    timeout: bool
    infra_failure: bool
    printable: bool

CONSTRAINT_HINTS = {
    "Do not change CSV column order in ledger export": "ledger",
    "Stamp timezone must be UTC, not the server locale": "UTC",
}

def git_diff_stat(repo: Path) -> tuple[int, list[str]]:
    diff = subprocess.check_output(["git", "-C", repo, "diff", "--numstat", "HEAD"], text=True)
    files, loc = [], 0
    for line in diff.splitlines():
        parts = line.split("\t")
        if len(parts) != 3:
            continue
        added, deleted, path = parts
        if added == "-" or deleted == "-":
            continue
        loc += int(added) + int(deleted)
        files.append(path)
    return loc, files

def constraints_hit(repo: Path, constraints: list[str]) -> int:
    blob = subprocess.check_output(["git", "-C", repo, "diff", "HEAD"], text=True)
    hits = 0
    for c in constraints:
        needle = CONSTRAINT_HINTS.get(c, "")
        # Toy check: real suites should use dedicated assertion files.
        if needle and needle in blob:
            hits += 1
    return hits

def score_run(task: dict, repo: Path, hidden_pass: bool, timeout: bool, infra: bool) -> TaskScore:
    loc, files = git_diff_stat(repo)
    allowed = set(task["allowed_files"])
    excess = [f for f in files if f not in allowed]
    hits = constraints_hit(repo, task["hidden_constraints"])
    locality_ok = (len(files) <= task["max_files_touched"]) and (not excess) and (loc <= task["max_diff_loc"])
    printable = (not timeout) and (not infra) and hidden_pass and locality_ok and hits == len(task["hidden_constraints"])
    return TaskScore(
        task_id=task["id"], origin=task["origin"], leakage_risk=task["leakage_risk"],
        hidden_pass=hidden_pass, constraint_hits=hits, constraint_total=len(task["hidden_constraints"]),
        files_touched=len(files), files_allowed=len(allowed), excess_files=len(excess),
        diff_loc=loc, max_diff_loc=task["max_diff_loc"], locality_ok=locality_ok,
        timeout=timeout, infra_failure=infra, printable=printable,
    )

Read the last field slowly. printable is true only when the hidden tests pass, the constraints hit, the diff stayed inside the fence, and the run was not an infra event. That is stricter than “pytest went green.” It is also the only boolean you should ever paste into a channel.

For a set, publish a table with one row per origin × leakage_risk cell. Counts, not rates, until every cell has a pre-declared minimum n. If a cell has three tasks, you write “3/3 printable” or “1/3 printable.” You do not write 33% as if it were a property of the agent.

Controls that keep the table from becoming marketing

Wall-clock, tool steps, and file fences belong in the run record. So does the judge version. If you change hidden tests, you bump the judge, not the agent name.

# run-record fields you store next to the diff
run_id: 2026-09-20T14-02Z-PAY-1847-a3
task_id: PAY-1847
judge_version: schema-1.2
agent_label: free-tier-runner    # never a model marketing name unless you logged it
timeout_sec: 180
tool_step_cap: 40
hidden_tests_mounted_after_exit: true
seed: 17
exclusion: null

Equalize what you compare. A 180-second cap on one agent and a 15-minute cap on another is a different exam. A tool cap of 40 versus unlimited bash is a different exam. You may still run both. You may not average them.

Seed the sampling. Free-tier decoding is noisy. One lucky pass on a three-task cell is not a ranking. Repeat the same frozen IDs. Report the repeats as repeats, not as extra tasks.

A decision table, not a leaderboard

Use this table before anyone quotes a number outside the team that built the suite.

Question	If no	What you may publish
Does every row have origin and leakage_risk?	Stop	Protocol draft only
Are hidden tests mounted after the agent exits?	Stop	Qualitative notes
Are infra failures in exclusions.jsonl instead of scored as misses?	Stop	Exclusion log
Do you report per-stratum counts, not one blend?	Stop	Per-task diffs
Is n per cell pre-declared and met?	Stop	“insufficient n”
Did both agents share timeout, tool cap, and judge version?	Stop	Separate run cards
Did the patch pass locality and hidden constraints?	Mark unprintable	Diff + failing field
Did all of the above hold?	—	Counts, with protocol hash

The last row is the only path to a number. Even then the number is a count attached to a protocol hash, not a brand claim. If you cannot name the hash, you cannot name the score.

Where a free runner actually helps

You can execute this protocol without a paid cluster. The dataset is YAML. The scorer is a Python file and git diff. The expensive part is the agent loop, and that is where a free-tier host is useful as a runner, not as a source of glory numbers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already need a place to park the runner, MonkeyCode’s free model access and free server option can host the agent loop and the after-exit judge. Label those runs runtime: free-tier in the record above. Do not mix them with a different host, a different timeout, or a different tool cap and then sort the rows as if the runtime were noise.

The protocol does not care which vendor paid for the tokens. It cares that you wrote the runtime down. A free box that keeps the mount honest is worth more than a paid box that lets the agent read tests/.

Limitations

The constraint check in the sample scorer is a toy. String-matching “UTC” in a diff will both false-pass and false-fail. Production suites should encode each hidden constraint as an assertion file the judge runs after the agent is gone. Do not ship the sample CONSTRAINT_HINTS map into CI and call it science.

Locality bounds punish legitimate refactors. A three-line bug that truly requires a fourth file will look like a miss. That is acceptable for a repair benchmark. It is the wrong metric for a “modernize this module” benchmark. Split those into two sets. Do not reuse max_files_touched: 2 as a moral value.

Provenance tags are only as good as the person who set them. If you copy a well-known public issue and mark leakage_risk: low because you renamed a function, you have laundered the task. When in doubt, mark high and keep the cell separate.

This protocol says nothing about user experience, latency under load, or safety. A printable patch can still be rude, slow, or leak a secret in a log line you never asserted on. Add those oracles if they matter. Do not pretend hidden pytest covers them.

Who should not use this

Skip this if you need a press ranking this week. The schema shrinks your set and delays the first number. That is incompatible with a launch blog.

Skip this if your tasks have no hidden tests and no constraints worth encoding. A prompt-only coding quiz is a different instrument. Forcing the YAML onto it will not make it a patch benchmark.

Skip this if you cannot freeze IDs. Teams that rewrite failed tasks until the agent passes are debugging the suite, not the agent. There is a place for that work. It is not evaluation.

Skip this if you will not keep infra rows in the exclusion log. A free server that restarts mid-run will manufacture misses. If you cannot label those, you should not rank anything that ran there.

What you do on Monday

Pick ten closed tickets. Fill the YAML. Hide the tests. Run one agent twice on the same IDs with the same cap. Print TaskScore rows. If you cannot explain a single printable: false from the fields, the schema is doing its job and the Slack number was never real.

Keep the protocol hash next to the table. When someone asks “what did it score?”, you send the table, the exclusions, and the hash. You do not send 81%.

If you try the runner path on MonkeyCode’s free model access and free server option, use it to keep the mount and the record honest. The score still has to survive the decision table. The host does not get a vote.

Comments

[howcani howcani]

I pasted your scorer into a scratch repo — one allowed file, one clean one-line diff, tests passing, no timeout, no infra — and ran four cases through score_run, touching nothing but the task dict. Three of the four print the same thing, and one of those three is not the agent's fault:

case	constraint_hits	hidden_pass	timeout	printable
as written in your YAML	1/1	true	false	true
same run, one constraint sentence reworded	0/2	true	false	false
same run, hidden test fails	1/1	false	false	false
same run, the box timed out	1/1	true	true	false

Row 2 is the one I'd fix before anyone mounts this. CONSTRAINT_HINTS.get(c, "") returns an empty needle for any sentence that isn't in the map, and an empty needle can never hit — so a constraint whose prose you reworded, or a new one you added to a task, makes that task permanently unprintable. Not a false pass and not a false fail, which is the toy behaviour you already flag: a silent configuration change that removes a task from ever being printable, with constraint_hits: 0-ish as its only trace. In the table it reads as an agent that violated a constraint. It is an operator who edited a sentence.

Cheapest fix, since the failure belongs to the map and not to the diff: index by a stable constraint id (utc_not_locale) and keep the prose as a display field, or make the lookup CONSTRAINT_HINTS[c] so an unmapped sentence raises at task-load time instead of after a run. Either way the failure lands before the run, which is the only time it is cheap.

Rows 2, 3 and 4 are also three different events under one boolean. Row 3 is a miss. Row 4 is a non-run — you say so yourself in the decision table ("infra failures in exclusions.jsonl instead of scored as misses"), and your Limitations note that a free box "will manufacture misses". But a timeout still produces a TaskScore with printable: false, and TaskScore has no exclusion field, so the object you aggregate cannot say "this row is not in the denominator" — that fact lives only in the run record next to it. Which means 1/3 printable over a cell where one box restarted is the same string as 1/3 where the agent failed twice, and that boolean is the one field you tell people to paste into a channel.

So outcome: pass | miss | not_run on the score object, and print counts as x/n with n = runs that ran. Your max_files_touched gap deserves the same treatment — a repair that needs a fourth file is a non-run under a repair benchmark, and your Limitations already say it must not be read as a miss.

One small thing while you are in there: exclusion: null in the run record and exclusions.jsonl are two carriers for one fact. The test I would apply is whether the excluded set can be recomputed from the run records alone — if the criterion exists only in the person's head at the time, then the log is a note rather than a denominator.