Count the Empty Cells Before You Quote a Pass Rate

A pass rate is an average over a grid you never drew. That's the whole problem. Ninety-four percent sounds like a verdict, but it's really a summary of whatever happened to be sitting in your case file that afternoon, and the file has holes in it you've never looked at.

I stopped trusting my own numbers the day I printed the grid instead of the total. Four of fifteen cells were empty. Not "thin" — empty. Nobody had ever sent the model an empty input with a tool-call contract, because writing that case is boring and it felt redundant. It wasn't redundant. It was the one place the schema silently returned a fabricated argument.

So here's the workflow I now run before any eval number leaves my laptop: draw the taxonomy, audit the grid, gate the commit. More cases is not the fix. Coverage first, volume second.

Draw the grid before you write the cases

The taxonomy is a small YAML file and it should live in git next to the cases it describes. Two axes are enough to start: what goes in, and what contract comes out. The failure classes are a third axis you tag cases with, not a cell dimension.

# taxonomy.yaml
axes:
  input_shape:     [empty, short, long_context, unicode, hostile]
  output_contract: [free_text, json_object, tool_call]
  failure_class:   [refusal, truncation, schema_violation, invented_field]

Five by three is fifteen cells. Two cases per cell is thirty cases, which is a completely reasonable afternoon. Compare that to the thousand-case dump people paste together and feel good about — a thousand cases that all land in four cells is just one cell wearing a costume.

Why two per cell and not one? Because one case can't distinguish "the model fails here" from "this prompt was cursed." Two gives you a coin-flip check. If the two disagree, you have a finding, and it's a finding about the case, not the model.

The auditor

This is the whole artifact, about fifty lines. It reads JSONL, refuses to guess, and exits non-zero when the grid has holes.

#!/usr/bin/env python3
"""coverage.py - audit eval-set coverage before you trust a pass rate."""
import argparse, collections, itertools, json, sys
import yaml


def load_cases(path):
    out = []
    with open(path, encoding="utf-8") as fh:
        for lineno, raw in enumerate(fh, 1):
            raw = raw.strip()
            if not raw or raw.startswith("#"):
                continue
            rec = json.loads(raw)
            rec["_lineno"] = lineno
            out.append(rec)
    return out


def main(argv=None):
    p = argparse.ArgumentParser()
    p.add_argument("cases", help="JSONL, one case per line")
    p.add_argument("--taxonomy", default="taxonomy.yaml")
    p.add_argument("--min-per-cell", type=int, default=2)
    p.add_argument("--gate", action="store_true", help="exit 1 on any hole")
    args = p.parse_args(argv)

    tax = yaml.safe_load(open(args.taxonomy, encoding="utf-8"))
    xs = tax["axes"]["input_shape"]
    ys = tax["axes"]["output_contract"]
    known = set(tax["axes"]["failure_class"])

    grid, off_taxonomy = collections.Counter(), []
    for rec in load_cases(args.cases):
        tags = rec.get("tags", {})
        x, y = tags.get("input_shape"), tags.get("output_contract")
        cell_ok = x in xs and y in ys
        if not cell_ok:
            off_taxonomy.append((rec["_lineno"], x, y))
        grid[(x, y)] += 1
        fc = tags.get("failure_class")
        if fc not in known:
            off_taxonomy.append((rec["_lineno"], "failure_class", fc))

    cells = list(itertools.product(xs, ys))
    empty = [c for c in cells if grid[c] == 0]
    thin = [c for c in cells if 0 < grid[c] < args.min_per_cell]
    filled = len(cells) - len(empty)

    print(f"cells: {filled}/{len(cells)} filled  "
          f"coverage: {filled / len(cells):.0%}  "
          f"min_per_cell: {args.min_per_cell}")
    print(f"{'input_shape':<14}" + "".join(f"{y:>16}" for y in ys))
    for x in xs:
        print(f"{x:<14}" + "".join(f"{grid[(x, y)]:>16}" for y in ys))

    for lineno, a, b in off_taxonomy:
        print(f"  ! line {lineno}: off-taxonomy tag {a}/{b}")
    for x, y in empty:
        print(f"  hole: {x} x {y}")
    for x, y in thin:
        print(f"  thin: {x} x {y} = {grid[(x, y)]}")

    if args.gate and (off_taxonomy or empty or thin):
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

A case looks like this. Note that the tags are the load-bearing part, and the prompt is almost an afterthought.

{"id":"json-0042","prompt":"Return a JSON object describing a city that is not mentioned anywhere in this prompt.","tags":{"input_shape":"hostile","output_contract":"json_object","failure_class":"invented_field"}}

What a failing audit looks like

Here is a run against a synthetic fixture I kept while testing the script itself. These numbers are fabricatd seed data, not measurements of any model — I'm showing you the shape of the report, not a benchmark result.

$ python coverage.py fixture.cases.jsonl --gate
cells: 11/15 filled  coverage: 73%  min_per_cell: 2
input_shape            free_text     json_object      tool_call
empty                          2               1              0
short                          4               3              2
long_context                   2               2              0
unicode                        3               2              0
hostile                        0               2              2
  hole: empty x tool_call
  hole: long_context x tool_call
  hole: unicode x tool_call
  hole: hostile x free_text
  thin: empty x json_object = 1
$ echo $?
1

Look at that column. tool_call is only exercised by short and hostile input. So any claim I make about tool calling is really a claim about two input shapes, and the moment someone passes a truncated document or an emoji-laden name through the same tool-calling path, I have nothing. The pass rate was never wrong. It was answering a smaller question than the one I asked.

The gate matters more than the report

The report is for you, once. The gate is for everyone after you. When the exit code blocks a PR that adds a duplicate case into an already-fat cell, you've turned "write more tests" from a vibe into a constraint. Add --min-per-cell 3 when you start making decisions that cost money, and watch how fast the grid changes shape.

Where free access actually fits

The expensive part of this workflow isn't the inference. It's the exploratory loop where you're writing cases, running them, and discovering that half your failure classes were mislabeled. That loop is bursty, low-stakes, and throws away most of its output. It's also exactly the loop where a metered API makes you timid — you write fewer hostile cases because each one feels like a purchase.

That's where I'd point you at MonkeyCode's free model access: it's useful for filling in empty cells, not for producing numbers you publish. The free server option is the companion piece, and it's the right host for the audit harness specifically — a short-lived job that reads JSONL, calls a model, writes a matrix. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Availability, limits, and supported models are the operator's to state, not mine to guess; check the current terms yourself before you build a habit around them.

A decision table, since this is where people overreach

Situation	Free model access	Free server option
Filling empty cells in a throwaway case file	Good fit	Good fit
Nightly coverage audit on a schedule	Good fit	Fine if the job is short and disposable
CI on every pull request	Use recorded responses, not live calls	No — don't hang your pipeline on someone else's box
Any latency or throughput claim	No	No
Numbers you intend to publish	No — measure the model you actually ship	No
Cases containing customer data	No	No

The pattern is simple. Free capacity is excellent for discovery and terrible for attestation. Keep them in separate files, or you'll mix them up at the worst moment.

Limits, and who should walk away

Coverage is not quality. A fully filled grid of fifteen cells tells you the taxonomy is exercised; it tells you nothing about whether your judge agrees with a human, and a single-judge setup will happily bless wrong output. The taxonomy is also a human artifact that drifts — six weeks from now unicode will need splitting into combining and rtl, and that's a git diff, not a bug.

If your eval set is already a maintained internal product with an owner and a schema, you don't need this; you need my script deleted. If you need latency numbers, or you're evaluating anything where the vendor's availability is a variable you can't control, keep your free-tier work in a sandbox and your claims in a lab.

So: how many cells in your grid are empty right now? You don't know, do you. That's the article. Draw the grid, run the audit, and let the exit code tell you what your percentage was hiding.