Leaderboards Are Not Evidence. Your Failure Corpus Is.

A user opens a document, asks the assistant to rename one section, and gets back a patch that edits a heading that stopped existing two saves ago. The prose was fine; the patch was confident, well-formed, and wrong. Notice where it actually broke: not in the model, but in the seam between a stale snapshot, a schema your validator never enforced, and a retry that replayed the old state. That seam is the layer I keep losing afternoons to, and no leaderboard on earth would have flagged it.

So here is the position I want to argue, and I will argue it plainly: a public model ranking is not evidence about your application, and treating it as evidence is how a team migrates a working feature onto a worse model with a better average. Leaderboards measure a mean over someone else's distribution of tasks, prompts, and tolerances. Your product fails on a specific, embarrassingly small set of inputs — a heading that moved, a tool call that returned 409, a JSON field that arrived as a string. The only instrument that measures those is a corpus you build from your own failures.

I have watched this play out in the last week's discourse too. There has been a run of posts arguing that executable scripts beat prose instructions for coding agents, and a benchmarking challenge wrap-up where the interesting part was the harnesses people built, not the final scores. Both point the same direction, and I think it holds for model choice as well: the harness is the asset, and the score is a side effect. Build the harness first, then argue about numbers.

What a failure corpus actually is

It is a directory of frozen incidents, each one small enough to read in a minute and boring enough to run a thousand times. Not a golden dataset of happy paths, and not a scraped eval set that someone else curated against someone else's product. Each case records the input, the response your system produced, the layer where it went wrong, and the exact check that would have caught it.

Here is the fixture format I use. Every field is something I can fill in from a ticket, a log line, or a reproduction that someone pasted into chat at 6pm.

{
  "id": "0142-retry-after-partial-write",
  "source": "link the ticket, log line, or repro — not just 'prod'",
  "layer": "state",
  "input": {
    "messages": [
      {"role": "system", "content": "You edit documents by returning patch.v3 JSON only."},
      {"role": "user", "content": "Rename 'Pricing' to 'Plans'. The doc snapshot is attached."}
    ],
    "snapshot_rev": 41
  },
  "expect": {"kind": "json_schema", "schema": "patch.v3"},
  "must_not": ["pricing", "create_heading"],
  "observed": {"http_status": 409, "note": "client retried against rev 39"}
}

The layer field matters more than it looks. When a case fails, my first question is whether the model, the transport, the state, or the validator is responsible, and I want that answer written down before I start guessing. Most of my early corpus entries turned out to be state bugs wearing a model costume, which is exactly the sort of thing a ranking will never tell you.

The runner, with a seam where the vendor goes

The design decision I would defend hardest is keeping the model call behind an adapter, so the corpus never imports a vendor SDK. I have been burned by this. The eval script grows a dependency, the dependency pins a version, and now the thing that was supposed to judge a migration is itself the riskiest upgrade in the repository.

# adapters/base.py
from typing import Protocol

class ModelAdapter(Protocol):
    name: str
    def complete(self, messages: list[dict], *, temperature: float = 0.0) -> dict:
        """Return {"text": str, "usage": dict, "latency_ms": int}."""

# adapters/http_compatible.py
import json, os, time, urllib.request

class HttpCompatible:
    """Transport is deliberately boring; the seam is the point."""
    def __init__(self):
        self.name = os.environ["MODEL_LABEL"]
        self.base = os.environ["MODEL_BASE_URL"].rstrip("/")
        self.key  = os.environ["MODEL_API_KEY"]

    def complete(self, messages, *, temperature=0.0):
        body = json.dumps({
            "model": os.environ["MODEL_ID"],
            "messages": messages,
            "temperature": temperature,
        }).encode()
        req = urllib.request.Request(
            f"{self.base}/chat/completions",
            data=body,
            headers={"Content-Type": "application/json",
                     "Authorization": f"Bearer {self.key}"},
        )
        start = time.monotonic()
        with urllib.request.urlopen(req, timeout=60) as resp:
            payload = json.loads(resp.read())
        return {
            "text": payload["choices"][0]["message"]["content"],
            "usage": payload.get("usage", {}),
            "latency_ms": int((time.monotonic() - start) * 1000),
        }

That request shape is a guess about your provider, and you should replace it with whatever the docs actually show you. The point is not the URL; the point is that swapping candidates becomes an environment variable, so nobody has to negotiate a re-architecture to test a model.

Now the scorer, which I keep dumb on purpose: parse, validate, and look for forbidden moves.

# run_corpus.py
import json, pathlib, statistics

CASE_DIR = pathlib.Path("corpus")

def load_cases():
    return [json.loads(p.read_text()) for p in sorted(CASE_DIR.glob("*.json"))]

def check(case, out):
    result, text = {}, out["text"]
    if case["expect"]["kind"] == "json_schema":
        try:
            doc = json.loads(text)
            result["parses"] = True
            result["schema_ok"] = validate(doc, case["expect"]["schema"])
        except json.JSONDecodeError:
            result["parses"] = False
            result["schema_ok"] = False
    for token in case.get("must_not", []):
        result[f"avoids:{token}"] = token.lower() not in text.lower()
    return result

def run(adapter, cases):
    rows = []
    for case in cases:
        out = adapter.complete(case["input"]["messages"])
        checks = check(case, out)
        rows.append({"id": case["id"], "layer": case["layer"],
                     "passed": all(checks.values()), "checks": checks,
                     "latency_ms": out["latency_ms"]})
    lat = sorted(r["latency_ms"] for r in rows)
    return {
        "adapter": adapter.name,
        "cases": len(rows),
        "pass_rate": round(sum(r["passed"] for r in rows) / len(rows), 4),
        "p95_latency_ms": lat[max(0, int(len(lat) * 0.95) - 1)],
        "failing_layers": sorted({r["layer"] for r in rows if not r["passed"]}),
        "rows": rows,
    }

validate() is whatever JSON Schema validator you already run in production, and if you do not have one, that absence is the real finding. The corpus is only as good as the check it enforces, so I would rather ship five cases with real validators than fifty with a substring match.

The loop I actually run looks like this, and it costs about as long as a coffee refill.

python -m corpus.check                      # fixtures parse; no model calls; safe for CI
MODEL_LABEL=current   python run_corpus.py > reports/current.json
MODEL_LABEL=candidate python run_corpus.py > reports/candidate.json
python compare.py reports/current.json reports/candidate.json

Run the baseline twice before you trust it, because temperature zero is not a promise. A case that flips between two identical runs is telling you to fix the validator or the state handling, not the prompt. I have deleted more clever prompts than I care to admit after noticing the failure was nondeterministic plumbing underneath.

The part where I refuse to promote on one number

A better pass rate is necessary and nowhere near sufficient, because a candidate can buy two points overall by breaking a layer you have never had to check. So I gate on layers, on determinism, and on whether the p95 latency fits inside the timeout budget the client already uses. And I gate on provenance, because a corpus written by the person who wants the migration approved will be a corpus that approves the migration.

Signal	Promote	Hold
Pass rate vs baseline	Equal or better, zero new failing layers	Plus two overall, but a new failure in state
Determinism	Same verdict across three runs	Flips between runs
p95 latency	Comfortably inside client timeout	Near timeout, retries masked by the harness
Case provenance	Each case traces to a ticket or log line	Cases authored to flatter the candidate

That table is opinionated, and I mean it that way. A failing layer is a bug you have not fixed yet, and no average is worth shipping one more of those.

Where a free tier and a free server honestly fit

Let me be precise about the gap, because it is not the cost of the model calls. The real friction is having no budget line for a candidate you might reject, so the baseline never gets run and the decision gets made in a meeting instead. That is the gap MonkeyCode's operator describes filling: free access to its models and a free server option, with the current offer quoted as ten million free tokens.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I have not benchmarked MonkeyCode here, and I cannot vouch for quotas, model lists, server specs, or how long any of it lasts. Treat those claims as operator-supplied and check the project's current terms before you plan a quarter around them. What I can say is that the workflow above is exactly the shape a free tier is good for: run the corpus batch somewhere that is not your laptop and not your CI minutes, keep the adapter seam so leaving is a one-line environment change, and never point production traffic at infrastructure whose terms you have not read.

Who should not use this

If your incidents contain customer data that cannot leave your controlled environment, do not paste them into any third-party service, free or paid, until someone who reads contracts says otherwise. If you need an uptime guarantee and a number you can page, a free server is an evaluation venue, not a place to serve users. And if your product has fewer than a handful of real failures recorded, you do not have a distribution yet — you have anecdotes, and a candidate that passes your forty cases has simply passed your forty cases.

Corpora also rot in a way that benchmarks do not. When the schema changes, half your cases silently test the old contract, so re-run them on every schema revision and delete the ones that no longer describe a failure you would care about. A stale corpus is worse than none, because it produces confident green checkmarks about a system you no longer run.

So which layer is your least stable handoff right now — the model, the transport, the state, or the validator? Send me one concrete failure and the response code it returned, and I will tell you what I would write as the fixture.

If you want a place to run that first batch without asking anyone for a card, MonkeyCode's free models and free server are a reasonable starting line.

mkdir -p corpus reports && cp your-first-incident.json corpus/0001.json
python -m corpus.check && echo "corpus is loadable; now make a candidate prove itself"

Comments

[Alex Shev]

A failure corpus earns its value when it is versioned, stratified, and hard to optimize against by accident. I’d keep a frozen evaluation slice for trend detection plus a rotating slice drawn from recent incidents, and publish per-category deltas rather than one aggregate score. That makes regressions actionable instead of merely disappointing.

[Eqvero]

The deceptive seam in financial extraction pipelines sits between schema validation and domain arithmetic. When parsing tables from SEC 10-Q filings, a model can return well-formed JSON that passes JSON Schema validation while failing basic accounting checks, like pulling an instant balance sheet fact instead of a three-month duration number.

We ended up pairing schema validation with deterministic assertion fixtures. Each failure fixture tests schema conformance, verifies that extracted line items sum to the reported statement total, and checks that duration metadata matches the filing period header. When evaluating candidate models, these cross-foot checks catch subtle omissions in period grounding that generic eval harnesses miss.