A coding-agent pass rate without a published retry cap is an incomplete measurement rather than a fair comparison. Hidden extra attempts inflate success in the same quiet way extra minutes inflate a closed-book exam score. Honest reporting therefore treats retry policy, wall-clock limits, and tool-trace identity as first-class dataset fields. Scores that omit those fields should be read as anecdotes rather than as comparative evidence.
Developers currently compare agentic coding tools with a single percentage that looks decisive in a README table. That number usually collapses several invisible knobs, including how many times the agent may regenerate a patch after a red test. Two systems can share a pass rate while one spent a single attempt and the other burned a silent loop of repairs. A methodology that does not freeze those knobs is closer to marketing copy than to a controlled experiment.
The proposed dataset is a sealed task pack rather than a live repository that changes under the runner. Each task record should include a unique identifier, a frozen prompt template, a hidden test digest, and an uneditable oracle command. The pack also needs an explicit retry ceiling and a wall-clock ceiling so that time and persistence are not left as folklore. Without those ceilings, later readers cannot reconstruct the labor that produced the headline success rate.
A practical pack can be a directory of JSON files that a runner refuses to load when required keys are missing. The following schema is a proposal, not a measured leaderboard, and it is meant to be boring on purpose. Keeping the file boring reduces the chance that a decorative field becomes a place to hide extra retries.
{
"task_id": "csv-split-001",
"prompt_template_id": "repair-v3",
"oracle_cmd": ["python", "-m", "pytest", "-q", "tests/test_oracle.py"],
"hidden_tests_sha256": "pending-local-hash",
"max_attempts": 2,
"wall_clock_s": 180,
"language": "python"
}
The metric set should stay small enough to print beside a percentage without an extra narrative apology. Primary success is oracle pass on a hidden suite after the final attempt, recorded as a boolean rather than a vibes score. Secondary fields include attempts_used, elapsed_ms, and a sha256 of the ordered tool-trace file so reviewers can detect silent reruns. A fourth field, scorecard_complete, is true only when every required control is present, which keeps incomplete runs from entering a table.
Controls matter more than the headline percentage because they explain why two similar percentages remain incomparable. The runner should disable workspace network writes, pin a seed when the task needs one, and stop hard after max_attempts. It should also refuse to grade against tests visible in the prompt, because visible tests turn the oracle into a hint sheet. Those controls do not make the published number large; they only make the number interpretable.
An analogy from manufacturing is useful here and keeps the argument from becoming a slogan. A factory can raise yield by reworking every unit until it passes, yet that yield is not the same as first-pass yield. Coding-agent scorecards that hide rework are reporting rework yield while labeling it first-pass yield. Readers who buy tools from such tables are comparing overtime shifts rather than comparing methods.
The original artifact below is an unexecuted reference runner that emits a scorecard or an error, never a lonely percentage. Operators can copy it into a working directory, replace the placeholder agent call, and keep the validation logic unchanged. The design goal is mechanical refusal, so incomplete metadata cannot masquerade as a published comparative score.
#!/usr/bin/env python3
"""Proposed retry-aware scorecard runner. Unexecuted reference, not a published result."""
from __future__ import annotations
import hashlib
import json
import subprocess
import tempfile
import time
from pathlib import Path
from typing import Any, Callable
REQUIRED_TASK_KEYS = (
"task_id",
"prompt_template_id",
"oracle_cmd",
"hidden_tests_sha256",
"max_attempts",
"wall_clock_s",
)
REQUIRED_SCORE_KEYS = (
"task_id",
"passed",
"attempts_used",
"max_attempts",
"elapsed_ms",
"wall_clock_s",
"prompt_template_id",
"hidden_tests_sha256",
"tool_trace_sha256",
"scorecard_complete",
)
class IncompleteScorecard(RuntimeError):
"""Raised when a run tries to publish a percentage without controls."""
def load_task(path: Path) -> dict[str, Any]:
task = json.loads(path.read_text(encoding="utf-8"))
missing = [key for key in REQUIRED_TASK_KEYS if key not in task]
if missing:
raise IncompleteScorecard(f"task missing fields: {missing}")
if int(task["max_attempts"]) < 1:
raise IncompleteScorecard("max_attempts must be at least 1")
return task
def hash_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def run_oracle(oracle_cmd: list[str], cwd: Path, timeout_s: int) -> bool:
try:
completed = subprocess.run(
oracle_cmd,
cwd=cwd,
check=False,
capture_output=True,
timeout=timeout_s,
)
except subprocess.TimeoutExpired:
return False
return completed.returncode == 0
def write_trace(trace_path: Path, events: list[dict[str, Any]]) -> str:
payload = json.dumps(events, separators=(",", ":"), sort_keys=True).encode("utf-8")
trace_path.write_bytes(payload)
return hash_bytes(payload)
def missing_score_fields(scorecard: dict[str, Any]) -> list[str]:
return [key for key in REQUIRED_SCORE_KEYS if key not in scorecard]
def run_task(
task: dict[str, Any],
workspace: Path,
agent: Callable[[dict[str, Any], Path, int], dict[str, Any]],
) -> dict[str, Any]:
started = time.monotonic()
deadline = started + int(task["wall_clock_s"])
events: list[dict[str, Any]] = []
passed = False
attempts_used = 0
for attempt in range(1, int(task["max_attempts"]) + 1):
remaining = deadline - time.monotonic()
if remaining <= 0:
events.append({"event": "wall_clock_exhausted", "attempt": attempt})
break
attempts_used = attempt
action = agent(task, workspace, attempt)
events.append({"event": "attempt", "attempt": attempt, "action": action})
oracle_timeout = max(1, int(remaining))
passed = run_oracle(list(task["oracle_cmd"]), workspace, oracle_timeout)
events.append({"event": "oracle", "attempt": attempt, "passed": passed})
if passed:
break
elapsed_ms = int((time.monotonic() - started) * 1000)
trace_hash = write_trace(workspace / "tool_trace.json", events)
scorecard = {
"task_id": task["task_id"],
"passed": passed,
"attempts_used": attempts_used,
"max_attempts": int(task["max_attempts"]),
"elapsed_ms": elapsed_ms,
"wall_clock_s": int(task["wall_clock_s"]),
"prompt_template_id": task["prompt_template_id"],
"hidden_tests_sha256": task["hidden_tests_sha256"],
"tool_trace_sha256": trace_hash,
"scorecard_complete": False,
}
scorecard["scorecard_complete"] = not missing_score_fields(scorecard)
if not scorecard["scorecard_complete"]:
raise IncompleteScorecard("refusing to emit an incomplete scorecard")
return scorecard
def validate_scorecard(scorecard: dict[str, Any]) -> None:
missing = missing_score_fields(scorecard)
if missing:
raise IncompleteScorecard(f"scorecard missing fields: {missing}")
if scorecard["attempts_used"] > scorecard["max_attempts"]:
raise IncompleteScorecard("attempts_used exceeds published retry cap")
if not scorecard["scorecard_complete"]:
raise IncompleteScorecard("scorecard_complete is false")
def demo_agent(task: dict[str, Any], workspace: Path, attempt: int) -> dict[str, Any]:
# Placeholder only: a real agent would write a patch here.
note = workspace / "attempt_log.txt"
note.write_text(f"{task['task_id']} attempt {attempt}\n", encoding="utf-8")
return {"wrote": str(note), "attempt": attempt}
if __name__ == "__main__":
sample = {
"task_id": "csv-split-001",
"prompt_template_id": "repair-v3",
"oracle_cmd": ["python", "-c", "raise SystemExit(1)"],
"hidden_tests_sha256": hash_bytes(b"sealed-oracle-bytes"),
"max_attempts": 2,
"wall_clock_s": 30,
"language": "python",
}
with tempfile.TemporaryDirectory() as tmp:
card = run_task(sample, Path(tmp), demo_agent)
validate_scorecard(card)
print(json.dumps(card, indent=2, sort_keys=True))
A companion test keeps the runner honest when someone later deletes a field to make a chart look cleaner. The test is small on purpose, because a methodology that needs a novel to explain its checks is already leaking discretion. Teams can extend the suite later, but they should not weaken the refusal path that blocks incomplete cards.
# test_scorecard.py — proposed unit checks, not a public leaderboard.
import json
from pathlib import Path
import pytest
from retry_scorecard import IncompleteScorecard, load_task, validate_scorecard
def test_load_task_rejects_missing_retry_cap(tmp_path: Path) -> None:
payload = {
"task_id": "x",
"prompt_template_id": "p",
"oracle_cmd": ["true"],
"hidden_tests_sha256": "abc",
"wall_clock_s": 10,
}
path = tmp_path / "task.json"
path.write_text(json.dumps(payload), encoding="utf-8")
with pytest.raises(IncompleteScorecard):
load_task(path)
def test_validate_rejects_over_budget_attempts() -> None:
scorecard = {
"task_id": "x",
"passed": True,
"attempts_used": 4,
"max_attempts": 2,
"elapsed_ms": 900,
"wall_clock_s": 30,
"prompt_template_id": "p",
"hidden_tests_sha256": "abc",
"tool_trace_sha256": "def",
"scorecard_complete": True,
}
with pytest.raises(IncompleteScorecard):
validate_scorecard(scorecard)
Commands stay ordinary so that the method can be reproduced on a modest local machine. The sequence below assumes the runner file is saved as retry_scorecard.py in the same directory as the tests.
python -m py_compile retry_scorecard.py
python retry_scorecard.py
pytest -q test_scorecard.py
sha256sum tool_trace.json 2>/dev/null || true
Those commands do not produce a vendor ranking, and that absence is the point of the workflow. A printed JSON object with scorecard_complete set to true is the only artifact this method is willing to call a result. A screenshot of a green model chat remains outside the measurement, even when the chat feels convincing to a human reviewer.
Free coding endpoints are useful here because they make it cheap to practice the bookkeeping before anyone argues about model quality. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which can host the sealed pack and the runner during rehearsal. The product is relevant only as a place to execute the controls, not as a substitute for the oracle or retry cap.
The numbers that leave this process are not marketing because they cannot be emitted without the knobs that usually hide in footnotes. Readers can disagree with a chosen max_attempts of two, yet they can see the choice and rerun with three if they wish. They can also reject a card whose tool_trace_sha256 does not match the committed trace file. Comparative disagreement then happens over explicit policy rather than over an unexplained headline percentage.
Limitations are part of the method rather than a closing apology paragraph for weak numbers. The runner does not prove production safety, and it does not estimate token cost, energy use, or long-horizon planning quality. It also assumes a deterministic hidden oracle, which many user-interface and flaky integration tasks do not have. Flaky tests will still flutter, and a dishonest operator can still edit traces before hashing if the workspace is not locked down.
This approach is a poor fit for several audiences that should pick a different evaluation style. Vendor marketing teams that need a single hero number will find the extra fields inconvenient, and that inconvenience is intentional. Classroom demos that celebrate any compiling patch should keep a teaching rubric instead of this scorecard. Teams under legal procurement rules need audited eval suites, contractually defined oracles, and human review, not a proposed script from an essay.
Current agent-writing discussions keep returning to systems that assume missing context and then spend extra loops repairing those assumptions. Retry-heavy scorecards reward exactly that habit by paying for success with hidden extra attempts. Publishing the cap makes the habit visible, which is a more useful conversation than another unexplained leaderboard bump. The headline percentage can stay, but it should arrive with the rework policy still attached.
People who want to try the bookkeeping can run the sample files on ordinary hardware and keep the JSON beside any later model claim. The invitation is only to publish fewer lonely percentages, not to treat a vendor name as a methodology. A complete scorecard remains a small JSON document, and that is the entire claim of this workflow.
Top comments (0)