Count Retries Before You Trust a Coding Score

A coding-agent score that hides retries, timeouts, and helper calls is closer to marketing than to measurement. Pass rate remains useful, but it becomes comparable only after an attempt budget and a retry ledger sit beside it. This methodology treats those controls as first-class fields, so a free-tier lab can be reproduced without inflating a lucky run. The numbers that survive this process are smaller, slower to collect, and far easier to defend.

Public talk about agentic coding still gravitates toward a single green check on a pull request. That check rarely records how many regenerations, tool loops, or stretched timeouts stood behind it. Two labs can therefore publish identical pass rates while spending very different amounts of model effort. A methodology that cannot show the spend is not a benchmark; it is a press release with a denominator.

Freeze the dataset before the first call

The proposed dataset is a small stratified bundle, built on purpose rather than scraped from a contest dump. Each task carries a frozen fixture directory, a public oracle, a wall-clock timeout, and a hard cap of k independent attempts. Stratification should mix repair, greenfield, and regression work so one easy cluster cannot dominate the mean. Tasks that leak the oracle into the prompt belong in a quarantine file, not in the scored set.

A practical directory layout keeps the contract obvious to anyone who later clones the repository. The following tree is a labeled proposal, not a claim about an existing public corpus.

bench/
  manifest.json
  tasks/
    repair-json-parser/
      prompt.md
      fixture/
      oracle/
      timeout_s.txt
    greenfield-csv-join/
      prompt.md
      fixture/
      oracle/
      timeout_s.txt
    regression-date-tz/
      prompt.md
      fixture/
      oracle/
      timeout_s.txt
  quarantine/

The manifest should pin identity and budget together so a moved fixture cannot silently change the score. Hashes belong beside names for that reason, and they should be computed before the first model call.

{
  "dataset_id": "retry-ledger-lab-2026-09",
  "k_attempts": 3,
  "timeout_s": 120,
  "temperature": 0.2,
  "hidden_helpers": false,
  "tasks": [
    {
      "id": "repair-json-parser",
      "stratum": "repair",
      "fixture_sha256": "replace-with-real-hash"
    }
  ]
}

Those fields are controls rather than decoration, and they belong in the scored record itself. Changing k, timeout, or temperature without a new dataset_id makes historical rows incomparable even when the pass rate looks stable.

Metrics that refuse to flatten spend

A single pass rate flattens several different outcomes into one cheerful percentage that hides spend. The proposed scorecard records pass_at_k, attempts used, timeout count, helper calls, and an oracle-leak flag for every task. Mean pass rate is then reported only with those companions, the same way a load test reports latency with error rate. Readers can see whether a win arrived on the first try or on the last allowed retry after a timeout was quietly raised.

The scorer below is an unexecuted reference implementation meant for a local JSONL retry ledger. It refuses to print a pass rate unless every row carries an attempt budget and a timeout.

# proposed scorer: retry_ledger_score.py
# Unexecuted reference. Replace hashes and paths before any real run.
from __future__ import annotations

import json
import sys
from collections import defaultdict
from pathlib import Path

REQUIRED = (
    "task_id",
    "attempt_index",
    "k_budget",
    "timeout_s",
    "elapsed_s",
    "passed",
    "timed_out",
    "helper_calls",
    "oracle_leak",
)


def load_rows(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text().splitlines():
        if not line.strip():
            continue
        row = json.loads(line)
        missing = [key for key in REQUIRED if key not in row]
        if missing:
            raise ValueError(f"incomplete ledger row: {missing}")
        rows.append(row)
    return rows


def score(rows: list[dict]) -> dict:
    by_task = defaultdict(list)
    for row in rows:
        if row["attempt_index"] >= row["k_budget"]:
            raise ValueError(f"attempt outside budget on {row['task_id']}")
        by_task[row["task_id"]].append(row)

    passed_tasks = 0
    attempts_used = 0
    timeouts = 0
    helpers = 0
    leaks = 0

    for task_id, group in by_task.items():
        group = sorted(group, key=lambda r: r["attempt_index"])
        attempts_used += len(group)
        timeouts += sum(1 for r in group if r["timed_out"])
        helpers += sum(r["helper_calls"] for r in group)
        leaks += sum(1 for r in group if r["oracle_leak"])
        passed_tasks += int(
            any(r["passed"] and not r["oracle_leak"] for r in group)
        )

    n = len(by_task)
    return {
        "tasks": n,
        "pass_at_k": (passed_tasks / n) if n else 0.0,
        "mean_attempts": (attempts_used / n) if n else 0.0,
        "timeout_rate": (timeouts / attempts_used) if attempts_used else 0.0,
        "helper_calls_per_task": (helpers / n) if n else 0.0,
        "oracle_leaks": leaks,
    }


if __name__ == "__main__":
    ledger = Path(sys.argv[1])
    result = score(load_rows(ledger))
    if result["oracle_leaks"]:
        raise SystemExit(f"oracle leak detected: {result}")
    print(json.dumps(result, indent=2))

A matching ledger row might look like the following example, taken from a failed first attempt. The numeric values are illustrative placeholders, not measured results from any hosted coding model.

{
  "task_id": "repair-json-parser",
  "attempt_index": 0,
  "k_budget": 3,
  "timeout_s": 120,
  "elapsed_s": 41.8,
  "passed": false,
  "timed_out": false,
  "helper_calls": 0,
  "oracle_leak": false,
  "temperature": 0.2
}

The command surface should be equally strict about writing exactly one ledger line per attempt. A wrapper that retries on failure without appending that line is a methodology bug, not a convenience.

# proposed local harness commands; unexecuted
python tools/run_task.py \
  --manifest bench/manifest.json \
  --task repair-json-parser \
  --attempt 0 \
  --ledger-out runs/ledger.jsonl

python tools/retry_ledger_score.py runs/ledger.jsonl

If the scorer exits because a row lacks k_budget or timeout_s, that failure is the point. Silent numeric defaults are how marketing scores sneak back into an otherwise careful laboratory notebook.

Controls that keep the lab from helping the model

Three process controls do most of the honesty work once the dataset itself is frozen. The value of k is chosen before any task runs and is never raised mid-lab because a favorite model looked weak. The wall-clock timeout is enforced by the harness process, not by the model's own stop condition. Hidden helpers stay off, including extra search tools, critic models, and unlogged pastes of failing tests into later prompts.

Temperature, seed if available, and fixture identity belong in the same ledger line as the pass bit. A rerun that changes any of those fields must write a new dataset_id rather than overwrite the old mean. Prompt caching across tasks should be disabled, or else recorded as a cache-hit flag on the ledger. A warm cache is an unpaid retry, and it will inflate pass_at_k without raising k.

Free-tier environments make these controls more important, not less, because retries are easy to hide. A cold process, a short timeout, and a visible attempt budget prevent the lab from borrowing production-grade retry loops. The methodology does not claim that free tiers are stronger; it claims they are easier to keep honest when the ledger is complete.

MonkeyCode is relevant here only as one place where that constrained coding lab can run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product offers free model access and a free server option for hosting a small harness and its ledger. Those availability claims are the only product facts used here; model names, quotas, hardware, and durability are omitted as unverified.

A free server is a reasonable home for the harness because the methodology wants process isolation more than peak throughput. Each attempt should start from the frozen fixture, write artifacts into a throwaway workspace, and append exactly one ledger line. That rhythm is closer to a lab notebook than to an always-on agent, which is why a disposable server fits the control plan. Readers who already keep local fixtures can load the same manifest on that free server and compare ledgers without changing the metric definitions.

Why the resulting numbers are not marketing

Marketing copy wants a large pass rate with a small footnote that nobody rereads later. This scorecard does the opposite by publishing a possibly modest pass_at_k next to mean attempts, timeout rate, and helper intensity. A method that reaches a high pass_at_k with mean attempts near one is one kind of object. A method that reaches the same rate only at k equals three, with frequent timeouts, is another.

The working analogy is highway fuel testing that still ignores idle time, cargo weight, and weather. A billboard ratio can look precise while the missing columns do the real explanatory work. Coding-agent scores without attempt accounting have the same shape: a clean ratio whose omitted retries carried the result. Once the ledger is public, other people can disagree with the metric weights, but they cannot pretend the retries never happened.

Limitations and who should skip this

The approach is slow by design and hostile to large, poorly labeled corpora that cannot be frozen. It does not estimate contamination against pretraining data, and it does not replace human review of diffs that merely match the oracle. Stratified sets of a few dozen tasks cannot rank general intelligence; they can only keep a local coding lab from lying to itself. Temperature control is incomplete on APIs that refuse seeds, so some variance will remain even with a perfect ledger.

Teams that need production latency SLAs should not treat this scorecard as a capacity plan. Vendor bake-offs that demand a single winner also should not use it, because the method keeps two numbers in view at once. Anyone unwilling to freeze fixtures, publish k, and log helper calls should skip the ceremony and call the work anecdotal. That blunt label is cheaper than a ledger whose columns will be ignored after the pass rate is copied.

The artifact is a contract that binds a pass rate to a retry ledger and a visible attempt budget. When those columns travel together, a free-tier coding lab can stay small, inspectable, and boring in the ways measurement requires. Boring columns are the point; they are what keep a coding score from turning back into marketing.