Find Fragile Cases With a Cross-Model Pass Matrix

A prompt that passes your eval on one model has told you almost nothing about how it behaves on the next one. The cheapest robustness signal you can collect is a pass matrix: the same golden cases run against several models, one pass vector per case, disagreement read before totals. The vector, not the percentage, is what shows you which cases your prompt only half handles.

Model churn is the constant in this work. Every few weeks another endpoint shows up with a different tokenizer, a different refusal posture, and a different opinion about what "concise" means. Teams pin one model, tune a prompt against it for a week, then switch providers when price or latency moves. That switch is a config change, it ships quietly, and nothing in the eval output looks unusual, because the eval only ever knew one model.

One grader, many adapters

The runner below keeps a single grader and pushes every provider difference into an adapter function. Cases live in a JSONL file so they can be diffed in review, and each case carries only checks a deterministic grader can evaluate. Inputs are truncated here for length; in your repository they should be the real payloads.

{"case_id":"refund-window","input":"How long do I have to return an opened item?","must_include":["14 days"],"must_not_include":["lifetime"]}
{"case_id":"invoice-json","input":"Summarize invoice 88 as JSON with total and currency.","json_fields":["total","currency"]}
{"case_id":"empty-cart","input":"My cart is empty, can I check out?","must_include":["cart is empty"]}

# sweep.py - standard library only
import hashlib, inspect, json, re, time
from datetime import datetime, timezone
from itertools import product

def norm(text):
    return re.sub(r"\s+", " ", text or "").strip().lower()

def grade(case, output):
    checks = []
    for needle in case.get("must_include", []):
        checks.append(("include:" + needle, norm(needle) in norm(output)))
    for needle in case.get("must_not_include", []):
        checks.append(("exclude:" + needle, norm(needle) not in norm(output)))
    if "json_fields" in case:
        try:
            payload = json.loads(output)
            ok = all(field in payload for field in case["json_fields"])
        except json.JSONDecodeError:
            ok = False
        checks.append(("json_fields", ok))
    return checks

GRADER_FP = hashlib.sha256(inspect.getsource(grade).encode()).hexdigest()[:12]

def fingerprint(obj):
    return hashlib.sha256(json.dumps(obj, sort_keys=True).encode()).hexdigest()[:12]

The adapter is the only provider-specific code in the whole harness, so adding a model means adding a config entry and one SDK call. The stub below is a contract, not a working client; it is deliberately unexecuted here, because credentials and SDK names are yours to choose.

def complete(model_id, prompt, params=None):
    """Implement per provider. Not executed in this article."""
    raise NotImplementedError

The sweep loop and its ledger

Each cell in the matrix stores its trial results, the last output, the case fingerprint, and the grader fingerprint. That combination is what makes a disagreement diagnosable a week later, when you no longer remember whether the prompt or the grader moved.

def sweep(models, cases, trials=3, pause=0.0):
    ledger = {}
    for model, case in product(models, cases):
        results = []
        for _ in range(trials):
            output = complete(model["id"], case["input"], model.get("params", {}))
            results.append(all(ok for _, ok in grade(case, output)))
            if pause:
                time.sleep(pause)
        ledger[(model["id"], case["case_id"])] = {
            "passed": all(results),
            "trials": results,
            "output": output,
            "prompt_fp": fingerprint(case),
            "grader_fp": GRADER_FP,
            "model_params": model.get("params", {}),
        }
    return ledger

def matrix(ledger, models, cases, path="matrix.csv"):
    lines = ["case_id," + ",".join(m["id"] for m in models) + ",class"]
    for case in cases:
        vector = "".join(
            "1" if ledger[(m["id"], case["case_id"])]["passed"] else "0"
            for m in models
        )
        label = "stable" if set(vector) == {"1"} else "blocked" if set(vector) == {"0"} else "fragile"
        lines.append(f"{case['case_id']},{','.join(vector)},{label}")
    open(path, "w").write("\n".join(lines) + "\n")
    return "\n".join(lines)

Requiring all trials to pass before a cell counts as a pass is a deliberate, blunt choice. It inflates the false-failure rate slightly and shrinks the false-pass rate a lot, which is the direction you want when the matrix is deciding where to spend human attention.

Reading a pass vector

A column of ones across every model means the case is stable enough to serve as a smoke test, and it earns a place in your fast suite. A column of zeros across every model is almost never a model problem; check the case text, then the grader, then the prompt, in that order. A mixed column is the interesting one, because it names a specific input where your instruction is ambiguous rather than hard.

Vector	Reading	First action
1111	stable	keep as smoke case
1101	fragile	inspect outputs, usually grader or ambiguity
0000	blocked	fix case, grader, then prompt

Triage order matters more than tooling here. Suppose refund-window produces 1101 and you discover the failing model answers "fourteen days" while the other three answer "14 days". The grader is doing substring matching on a normalized string, so the digit and the word are different tokens to it, and the disagreement was manufactured by your check rather than discovered in the model. The fix is a canonical form or a small regex alternation, applied once, and then a full re-run with a bumped grader fingerprint so the history stays honest.

If the outputs differ substantively instead, with one model adding a cheerful preamble and skipping the number, the ambiguity is in the prompt. Rewrite the instruction to require the number in the first sentence, re-run only the fragile column at first, and promote the case back into the full sweep once its vector is all ones. Blocked cases deserve the same treatment before they quietly rot in the suite.

What the sweep actually costs

The arithmetic is simple and worth doing before you start: models times cases times trials. Twenty-four cases, four models, and three trials is 288 calls, which is nothing on a metered API and still worth logging. A wider sweep is where free model access stops being a curiosity, because you can afford to keep several candidate models in the matrix instead of testing the one you already pay for.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Where that matters for this workflow is the boring operational half. MonkeyCode's free model access, as described by the operator, lets you keep a handful of adapters live in the matrix without a billing account, and the free server option means the sweep runner does not need its own provisioned machine. Confirm the current terms, model list, and any limits on the project's own pages before you build a routine around them, since availability terms change and this article cannot track them for you.

Limitations and who should skip this

Substring and JSON-key grading measures surface compliance, not correctness, so a green matrix is evidence about format and constraint adherence and nothing more. Three or four models are a sampled population rather than a ranking, and publishing cross-model leaderboards from twenty cases is statistically empty even when the table looks tidy. Free endpoints can throttle or change without notice, so cache outputs to disk keyed by model, case, and trial, and add backoff rather than retrying tightly.

Skip this if your product pins exactly one model with no fallback path, because the matrix will only confirm what you already assume. Skip it too if your grader is still unstable across repeated runs on one model, since you would then be measuring grader noise with extra steps. Fix the single-model baseline first, then widen.

Start with four models you would actually ship and fifteen cases you would actually argue about. The first fragile vector usually pays for the whole exercise.

If you want to run the sweep without a billing setup, put the runner on MonkeyCode's free server option and check the current model list before you wire up the adapters.