Don't Ship the Pass Rate. Ship the Failure Mix.

Last Tuesday you had two coding-agent traces for the same ticket. One was marked pass. One was marked fail. You almost stopped there.

The passing trace never opened the OpenAPI file. It guessed /v2/invoices, generated a client, and the unit test only asserted HTTP 200 against a mock the agent also wrote. The failing trace asked for the spec, then burned the rest of the run on three identical tool calls. If you average those outcomes into 50%, you learn nothing you can change in the prompt, the tools, or the review policy.

A headline percentage is a marketing object. A failure mix is an engineering object. This article is a method for producing the second one on a small, frozen eval you can rerun on a free server.

What a pass rate hides

Pass rate answers one question: did the scorer light up green. It does not tell you whether the agent assumed an API that does not exist, looped a tool, skipped a contract test, or got lucky because the fixture was too kind.

Vendors know this. A single number travels. A taxonomy does not. That is why you keep seeing leaderboards and almost never seeing the mix of why tasks died.

You do not need a giant public suite to fix that. You need a dataset you can hash, a scorer that is not another model, a budget you freeze before the run, and a label set that is coarser than "fail."

Freeze four controls before the first run

If any of these move between runs, you are not comparing agents. You are comparing weather.

1. Dataset hash. Pin the task pack: prompt files, fixtures, tests, and a SHA-256 over the directory. If a test changes, the card's dataset id must change.
2. Non-LLM scorer. Compile, unit tests, schema checks, or a diff against a golden file. An LLM-as-judge is a second system under test, not a metric.
3. Budgets. Cap wall-clock, output tokens, and tool calls before you launch. A "pass" that took 40 retries is a different product than a pass that took two.
4. Environment. Same container, same language toolchain, no network unless the task says so. Record the image digest on the card.

Write those four values down first. Then run. Not the other way around.

A taxonomy you can label in ten minutes

Keep the classes mutually exclusive at the primary label. Secondary tags are allowed. Eight buckets are enough for a coding agent:

Code	Name	You apply it when
PASS_CONTRACT	Real pass	Scorer green and the agent did not author the only passing test
PASS_WEAK	Lucky pass	Scorer green, but tests are mocks, snapshots, or agent-written
ASSUME_API	Invented surface	Calls, paths, or types not in the frozen spec
TOOL_LOOP	Repeated tool	Same tool + same args, N times, no new evidence
MISSING_ASK	Should have asked	Required file or credential was absent; agent guessed
SCOPE_SLIP	Extra work	Touched files outside the task allowlist
TIMEOUT	Budget kill	Hit wall-clock, token, or tool cap
SCORER_RED	Honest fail	Tests failed for a real assertion, no other bucket fits

You are not scoring intelligence. You are scoring the shape of the mistake. That shape is what tells you whether to add a spec-reading tool, a loop detector, or a "do not write tests" rule.

Illustrative mix, not a measurement I ran for you:

PASS_CONTRACT  2
PASS_WEAK      1
ASSUME_API     3
TOOL_LOOP      2
TIMEOUT        2

Two real passes and three invented APIs is a different system than seven passes. The average still looks like "50%." Your architecture decision does not.

Artifact: a frozen benchmark card

Treat the card as the published result. The percentage is a derived field, not the artifact.

{
  "card_version": "1",
  "dataset_id": "invoice-client-v3",
  "dataset_sha256": "replace-with-sha256-of-task-pack",
  "scorer": "pytest -q --tb=no",
  "budgets": {"wall_s": 180, "max_output_tokens": 4096, "max_tool_calls": 12},
  "env": {"image_digest": "sha256:replace-me", "network": false},
  "n_tasks": 10,
  "primary_counts": {
    "PASS_CONTRACT": 0,
    "PASS_WEAK": 0,
    "ASSUME_API": 0,
    "TOOL_LOOP": 0,
    "MISSING_ASK": 0,
    "SCOPE_SLIP": 0,
    "TIMEOUT": 0,
    "SCORER_RED": 0
  },
  "derived": {
    "headline_pass_rate": null,
    "contract_pass_rate": null
  }
}

headline_pass_rate is (PASS_CONTRACT + PASS_WEAK) / n_tasks. contract_pass_rate is PASS_CONTRACT / n_tasks. If those two diverge, you found the marketing gap. Publish both. Hide neither.

Numbered workflow you can actually run

1. Build a tiny pack, then hash it

Ten tasks is plenty if each one has a frozen spec, a hidden test you did not show the agent, and an allowlist of files. Put them in tasks/.

find tasks -type f | sort | xargs sha256sum | sha256sum
# record the digest as dataset_sha256

If you cannot hide at least one test per task, you cannot detect PASS_WEAK. Stop and add a test the agent cannot see.

2. Log traces as JSONL, not screenshots

Each line is one task. You need the scorer exit code, the file diff, and the tool log. Without the tool log you cannot label TOOL_LOOP or ASSUME_API.

{"task_id":"T04","exit_code":0,"tool_calls":[{"name":"read","args":{"path":"README.md"}}],"written_tests":false,"wall_s":41}

3. Classify with rules first, humans second

Start with deterministic rules. Humans only resolve leftovers. That keeps the method cheap and auditable.

#!/usr/bin/env python3
"""Classify coding-agent traces into a frozen failure mix.

This is a method script, not a published leaderboard.
Fill traces.jsonl with your own runs before you trust any count.
"""
from __future__ import annotations

import json
import collections
from pathlib import Path

TAXONOMY = (
    "PASS_CONTRACT",
    "PASS_WEAK",
    "ASSUME_API",
    "TOOL_LOOP",
    "MISSING_ASK",
    "SCOPE_SLIP",
    "TIMEOUT",
    "SCORER_RED",
)


def repeated_tools(calls: list[dict], n: int = 3) -> bool:
    window = []
    for call in calls:
        key = (call.get("name"), json.dumps(call.get("args", {}), sort_keys=True))
        window.append(key)
        if len(window) >= n and len(set(window[-n:])) == 1:
            return True
    return False


def classify(row: dict) -> str:
    calls = row.get("tool_calls") or []
    if row.get("timeout") or row.get("wall_s", 0) >= row.get("wall_cap", 10**9):
        return "TIMEOUT"
    if repeated_tools(calls):
        return "TOOL_LOOP"
    if row.get("assumed_api"):
        return "ASSUME_API"
    if row.get("required_missing") and not row.get("asked"):
        return "MISSING_ASK"
    if row.get("outside_allowlist"):
        return "SCOPE_SLIP"
    if row.get("exit_code") == 0 and row.get("written_tests"):
        return "PASS_WEAK"
    if row.get("exit_code") == 0:
        return "PASS_CONTRACT"
    return "SCORER_RED"


def main() -> None:
    rows = [json.loads(line) for line in Path("traces.jsonl").read_text().splitlines() if line.strip()]
    counts = collections.Counter(classify(r) for r in rows)
    n = len(rows) or 1
    card = {
        "n_tasks": len(rows),
        "primary_counts": {k: counts.get(k, 0) for k in TAXONOMY},
        "derived": {
            "headline_pass_rate": (counts["PASS_CONTRACT"] + counts["PASS_WEAK"]) / n,
            "contract_pass_rate": counts["PASS_CONTRACT"] / n,
        },
    }
    Path("benchmark_card.json").write_text(json.dumps(card, indent=2))
    print(json.dumps(card, indent=2))


if __name__ == "__main__":
    main()

Flag assumed_api with a boring checker: collect identifiers the agent called, subtract identifiers present in the frozen spec, and require a non-empty remainder. Do not ask a model if the call "looks right."

4. Compute two rates, then stop averaging them

headline_pass_rate  = (PASS_CONTRACT + PASS_WEAK) / n
contract_pass_rate  =  PASS_CONTRACT / n
assumption_share    =  ASSUME_API / n
loop_share          =  TOOL_LOOP / n

If assumption_share is high, your next patch is a spec tool or a deny-list on guessed paths. If loop_share is high, your next patch is a repeated-call breaker. Neither patch falls out of a 70% banner.

5. Rerun only when a control changes on purpose

Same card, new agent: keep dataset, scorer, budgets, environment. Same agent, new prompt: bump a prompt_id field, not the dataset id. If you change tests, you started a new benchmark. Say so.

Decision table: what the mix is telling you

Dominant bucket	Do not do	Do this
PASS_WEAK	Celebrate the pass rate	Hide tests; forbid agent-authored tests in the scorer
ASSUME_API	Add more samples of the same task	Inject the spec as a required read; fail closed on unknown paths
TOOL_LOOP	Raise the retry cap	Deduplicate tool args; stop at N repeats
MISSING_ASK	Fine-tune for "confidence"	Add an ask/abort tool and score it as success when the file is absent
TIMEOUT	Switch models first	Lower the task, or raise one budget and record it on the card
SCOPE_SLIP	Blame the model	Enforce an allowlist in the runner, not in the prompt

Read the table before you change models. Most of these are harness bugs dressed up as model quality.

Where a free model and a free server fit

You want the worker that runs the harness to be boring and restartable. A laptop that sleeps mid-eval contaminates wall-clock. A paid GPU box tempts you to inflate the suite until you cannot label failures by hand.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a place to park the runner, MonkeyCode's free model access and free server option are enough to execute this card against a small frozen pack. Use them as the worker, not as the judge. The taxonomy still comes from your tests and your tool log.

Keep the suite small on purpose. Ten labeled tasks with hidden tests beat two hundred unlabeled ones you will never inspect.

Limitations, and who should skip this

This method will not replace a large public software-engineering board. It will not give you a single number purchasing can paste into a slide. It will not tell you which model is "best" across languages you did not include.

Do not use it if you cannot hide tests from the agent. Do not use it if you refuse to log tool calls. Do not use it if the only scorer you have is another LLM. Do not use it as a live CI gate until two people independently label a held-out slice and agree on the primary bucket for most rows.

The script above is a classifier, not evidence. Empty traces.jsonl produces zeros. Zeros are not a result.

What to publish instead of a banner

Publish the card. Publish the dataset hash. Publish both pass rates. Publish the mix.

Then the next person who shows you a 90% agent has to answer a dull question: 90% of what failure shape, under which frozen budgets, against which hidden tests. If they cannot answer, you do not have a benchmark. You have a press quote.

If you run the card, keep the suite tiny enough that you can read every failing trace. The mix only works when you still look.