Label Infra Failures Before You Rank Coding Agents

You finish a weekend bake-off on a cheap shared box. Agent A sits at 62 percent. Agent B sits at 81 percent. You are already drafting the post.

Then syslog loads. Four of A's misses are Broken pipe during git clone. One is disk-full on /tmp. Two are pytest importing a stale .pyc after a previous job died mid-write. You did not measure models. You measured weather on the box.

This piece is a method for labeling those misses. The percentage you quote should be about agent skill. Not about who drew the noisy hour.

The mixing problem

Public threads keep saying models already outcode most developers. That claim might hold on a clean bench. It does not hold for a CSV that treats every non-zero exit as "the agent failed."

Shared boxes make the mix worse. Preempted CPUs, noisy neighbors, rate limits, and flaky disks land in the same column as a wrong patch. Publish that column and you are doing marketing in a lab coat.

You need four buckets. Only one belongs in the skill score.

Four buckets, one rule

Give every unsuccessful or unfinished run exactly one label before you aggregate.

1. MODEL — the agent produced a patch or a refusal, the harness stayed healthy, and the tests still fail for a task reason.
2. HARNESS — your runner, prompt template, grader, or sandbox config is wrong. The agent never had a fair shot.
3. INFRA — SSH drops, disk full, host OOM, provider 5xx, rate limits, wall-clock preemption. The task was not scored.
4. FLAKY_TEST — the same task commit fails or passes when you rerun with the agent disabled.

Rule: skill score equals MODEL successes divided by MODEL successes plus MODEL failures. INFRA, HARNESS, and FLAKY_TEST are censored. You report them. You do not fold them into "the model is worse."

That denominator change is the method. It is boring. It is also how you stop lying to yourself.

A 16-task holdout, not a trophy case

Do not scrape a public leaderboard and call it a dataset. Build a small holdout you can freeze.

Defend these selection rules:

1. Each task has a failing test you can run with the agent turned off.
2. The gold patch, if you have one, is absent from the prompt, the README, and comments the agent will read.
3. At least four tasks are negative controls that are already green. An agent that "fixes" them by rewriting working code is a grader bug.
4. At least four tasks need a multi-file edit. Single-function toys inflate MODEL success.
5. Record license, commit SHA, and a one-line reason the task belongs.

Store one JSON object per task.

{"id": "holdout-04", "repo": "example/billing", "sha": "a1b2c3d", "test_cmd": "pytest -q tests/test_prorate.py", "expect_fail_without_patch": true, "negative_control": false, "multi_file": true, "license": "MIT"}

Sixteen tasks is enough to debug the method. It is not enough to crown a winner. Print that caveat in the same paragraph as any rate.

Metrics that survive a noisy box

You will want a single pass rate. Publish the split instead.

• eligible_n: rows labeled MODEL
• resolved_rate: MODEL successes / eligible_n
• infra_censor_rate: INFRA / scheduled_n
• harness_error_rate: HARNESS / scheduled_n
• flaky_rate: FLAKY_TEST / scheduled_n
• time_to_first_green_s: MODEL successes only
• bytes_changed: git diff --stat, so silent no-ops cannot count as wins

If infra_censor_rate is high, you do not have a model comparison. You have a hosting diary. Stop. Fix the box or throw the batch out.

Controls you log on every run

Number them so a reviewer can audit you.

1. Harness commit SHA and container digest.
2. Task-suite SHA of the JSONL file.
3. Clock: start, end, timezone, NTP status.
4. Host fingerprint: uname -a, load average, free disk, memory.
5. Reachability of git and the model endpoint before the first prompt.
6. Retry policy: zero retries on the published number. Retries belong in an appendix.
7. Decoding settings as the API returned them, not as you hoped.
8. A probe that must fail: empty patch against a red test. If it "passes," your grader is broken.

Drop this probe into the job preamble.

#!/usr/bin/env bash
set -euo pipefail
{
  echo "ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "host=$(hostname)"
  uname -a
  uptime
  df -h / /tmp
  free -m || true
  git -C "$HARNESS_DIR" rev-parse HEAD
  sha256sum "$SUITE_JSONL"
  curl -sS -o /dev/null -w "endpoint_http=%{http_code}\n" --max-time 10 "$MODEL_HEALTH_URL" || echo "endpoint_http=down"
} | tee "$RUN_DIR/fingerprint.txt"

If free disk is already tight, label the whole batch INFRA. Do not start.

Artifact: classify the log

The script below is a proposal. Tune the patterns on your harness. Read the borderline rows yourself.

#!/usr/bin/env python3
"""Label one coding-agent run. Proposal — tune patterns on your harness."""
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

INFRA = [
    r"Broken pipe",
    r"Connection reset by peer",
    r"No space left on device",
    r"Killed",
    r"Out of memory",
    r"HTTP 429",
    r"HTTP 5\d\d",
    r"deadline exceeded",
    r"Could not resolve host",
]
HARNESS = [
    r"ModuleNotFoundError: No module named 'harness'",
    r"ERROR: file not found: pytest",
    r"unknown flag:",
    r"sandbox config",
]

def first_match(text: str, patterns: list[str]) -> str | None:
    for p in patterns:
        if re.search(p, text, re.I):
            return p
    return None

def classify(log: str, tests_without_agent_pass: bool | None, tests_after_pass: bool | None) -> dict:
    if tests_without_agent_pass is True and tests_after_pass is True:
        return {"label": "FLAKY_TEST", "reason": "suite already green with agent disabled"}
    if (m := first_match(log, INFRA)):
        return {"label": "INFRA", "reason": m}
    if (m := first_match(log, HARNESS)):
        return {"label": "HARNESS", "reason": m}
    if tests_after_pass is True:
        return {"label": "MODEL", "reason": "eligible success", "success": True}
    if tests_after_pass is False:
        return {"label": "MODEL", "reason": "eligible miss", "success": False}
    return {"label": "HARNESS", "reason": "grader produced no boolean"}

def main() -> None:
    run_dir = Path(sys.argv[1])
    log = (run_dir / "combined.log").read_text(errors="replace")
    meta = json.loads((run_dir / "meta.json").read_text())
    result = classify(
        log,
        meta.get("tests_without_agent_pass"),
        meta.get("tests_after_pass"),
    )
    result["run_id"] = meta.get("run_id")
    (run_dir / "label.json").write_text(json.dumps(result, indent=2) + "\n")
    print(result["label"], result["reason"])

if __name__ == "__main__":
    main()

Keep meta.json tiny:

{
  "run_id": "2026-09-16T04-11Z-holdout-04",
  "agent": "A",
  "task_id": "holdout-04",
  "tests_without_agent_pass": false,
  "tests_after_pass": false
}

Aggregate only after every row has a label.

# aggregate.py — proposal
import json
from collections import Counter
from pathlib import Path

rows = [json.loads(p.read_text()) for p in Path("runs").glob("*/label.json")]
c = Counter(r["label"] for r in rows)
model = [r for r in rows if r["label"] == "MODEL"]
wins = sum(1 for r in model if r.get("success"))
eligible = len(model)
print("scheduled", len(rows))
print("counts", dict(c))
print("resolved_rate", None if eligible == 0 else round(wins / eligible, 3))
print("infra_censor_rate", round(c["INFRA"] / len(rows), 3) if rows else None)

Read that printout before you open a charting tool.

Decision table

Symptom	First check	Label	In skill score?
SSH / clone / 5xx / 429 / disk / OOM	fingerprint.txt	INFRA	No, censor
Grader exception, missing pytest, bad cwd	harness SHA	HARNESS	No, fix then rerun
Agent disabled, tests already pass	negative control	FLAKY_TEST or bad task	No
Patch applied, harness healthy, tests red	diff + log	MODEL miss	Yes
Patch applied, tests green, negative control untouched	diff + log	MODEL hit	Yes
Agent rewrites a green negative control	diff	HARNESS or grader	No until explained

If two labels could apply, pick the earlier failure in wall-clock order. An OOM before the model returns is INFRA, not MODEL.

Where free model access and a free server fit

You do not need a fancy cluster to practice this. You need logs, a frozen suite, and a box that can finish a batch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. That pair helps when you are debugging the taxonomy: run the same 16 tasks, keep the fingerprint file, and see whether your INFRA rate is the method or the host. It is not a reason to skip labels. If the free server is noisy that day, the honest result is a high infra_censor_rate and no ranking.

Why these numbers are not marketing

Marketing wants one percentage and a winner. A lab report wants a denominator you can defend.

You do not get to say models outcode most developers from 16 tasks on a shared box. You do not get to say it when a third of the misses are SSH.

Publish this block, or do not publish a number:

• suite SHA and task-selection rules
• eligible_n and scheduled_n
• resolved_rate on MODEL rows only
• infra, harness, and flaky rates
• retry policy (zero for the headline)
• a sentence that the set is a method check, not a ranking of humanity

If a vendor quotes a score without those lines, treat it as copy. Not as measurement.

Limitations, and who should skip this

The regex classifier will mislabel. A model that prints Connection reset by peer inside a fixture can look like INFRA. Read those rows.

Four buckets are coarse. A wrong test pin is not the same as a confused planner. You can add sublabels later. Do not add them in the same week you still mix SSH into skill.

Free hosts can censor so many runs that eligible_n collapses. Then the method worked. It told you not to rank anyone.

Do not use this approach if you need production SLAs, if you cannot freeze the suite, if your tests hit the live network, or if your goal is a launch post. Do not compare agents that saw different prompts, timeouts, or days without saying so.

Skip it if you will not look at diffs. Labels without diffs are another costume.

A closing check

Before you quote a figure, open one failing log. Write the bucket on paper.

If you hesitate, the number is not ready.