A Model-Backed Test That Passed Once Is a Sample Size of One

A single green run of a model-backed test tells you almost nothing. It is one draw from a distribution you have never measured. Until you know that distribution, your merge button is a coin flip with extra steps.

Here is the fix in one sentence: sample each model-backed fixture N times, write a ledger, and let the merge rule read the ledger instead of the last exit code.

This post gives you the fixture contract, a reference sampler, a decision table, and two hooks — a cheap one locally, a strict one in CI.

Why reruns make the problem worse

Most teams handle a red model test by rerunning it. It goes green. Everyone moves on. Repeat that three times and you have trained the team to ignore the check.

There are two failure modes and they look identical in the UI. A real regression that failed once and passed on retry. Or a genuinely unstable case that passed this time and will fail after lunch. Reruns cannot separate them, because a rerun only gives you a second sample you throw away.

So stop rerunning to green. Start recording samples, then decide.

Step 1: Make the fixture assert on invariants, not prose

Most false flakes are self-inflicted. If your assertion is "output equals this paragraph," every run is a coin flip. Assert on structure, ranges, and required fields instead.

# tests/model/cases/summarize_changelog.yaml
id: summarize_changelog
input: fixtures/changelog.txt
expect:
  format: json
  required_keys: [breaking, added, removed]
  max_items_per_key: 5
  forbid_substrings: ["I cannot", "As an AI"]
  max_output_chars: 1200

Four rules keep this honest. Assert JSON shape, not wording. Assert counts and bounds, not exact strings. Never assert on whitespace or markdown formatting. And keep one "golden" case per feature, not per fixture — golden cases are for humans to read, not for the gate.

Now write each case outcome to a log so a sampler can read it.

# tests/model/conftest.py
import json, os
from pathlib import Path
import pytest

LOG = Path(os.environ.get("MODEL_RUN_LOG", "artifacts/run.jsonl"))
RUN_ID = os.environ.get("MODEL_RUN_ID", "local")

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when != "call":
        return
    LOG.parent.mkdir(parents=True, exist_ok=True)
    with LOG.open("a") as fh:
        fh.write(json.dumps({"run": RUN_ID, "case": item.nodeid, "passed": report.passed}) + "\n")

Step 2: Sample, tally, and write a ledger

The sampler below is a reference implementation. Trim it, rename it, and run it against your own suite — the numbers it prints will be yours, which is the only kind worth having.

#!/usr/bin/env python3
# tools/sample_model_tests.py
import hashlib, json, os, subprocess, sys
from collections import defaultdict
from pathlib import Path

RUNS = int(os.environ.get("SAMPLE_RUNS", "5"))
FLAKY_BUDGET = int(os.environ.get("FLAKY_BUDGET", "0"))
CASES = Path("tests/model/cases")
LEDGER = Path("artifacts/model-ledger.json")


def fixture_digest() -> str:
    h = hashlib.sha256()
    for p in sorted(CASES.glob("*.yaml")):
        h.update(p.name.encode())
        h.update(p.read_bytes())
    return h.hexdigest()[:12]


def main() -> int:
    digest = fixture_digest()
    log = Path("artifacts/run.jsonl")
    log.unlink(missing_ok=True)

    for i in range(RUNS):
        env = {**os.environ, "MODEL_RUN_ID": f"{digest}-{i}", "MODEL_RUN_LOG": str(log)}
        subprocess.run([sys.executable, "-m", "pytest", "tests/model", "-q"], env=env, check=False)

    tally = defaultdict(lambda: [0, 0])  # case -> [passes, runs]
    for line in log.read_text().splitlines():
        row = json.loads(line)
        tally[row["case"]][1] += 1
        tally[row["case"]][0] += int(row["passed"])

    cases, flaky = {}, []
    for case, (passes, runs) in sorted(tally.items()):
        rate = passes / runs
        state = "stable-pass" if rate == 1 else "stable-fail" if rate == 0 else "flaky"
        if state == "flaky":
            flaky.append(case)
        cases[case] = {"passes": passes, "runs": runs, "pass_rate": round(rate, 2), "state": state}

    ledger = {"fixture_digest": digest, "runs_per_case": RUNS, "flaky": flaky, "cases": cases}
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    LEDGER.write_text(json.dumps(ledger, indent=2) + "\n")
    print(json.dumps({"fixture_digest": digest, "runs_per_case": RUNS, "flaky": flaky}, indent=2))

    if len(flaky) > FLAKY_BUDGET:
        print(f"BLOCK: {len(flaky)} flaky case(s) exceed budget {FLAKY_BUDGET}", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

Two details matter more than they look. The fixture_digest ties a ledger to an exact set of fixtures, so a stale ledger cannot silently bless new cases. And the budget is a number you set deliberately, not a vibe.

Step 3: Turn the ledger into a merge rule

A pass rate is only useful if it maps to an action. This table is the whole policy — copy it into your PR template so reviewers see the same rule the gate uses.

Ledger state	Pass rate	Merge decision
stable-pass	1.00 over 5 runs	Merge. Keep it as a required check.
stable-fail	0.00 over 5 runs	Block. This is a regression, not noise.
flaky, mostly passing	≥ 0.60	Quarantine to nightly, open a ticket, block only if the case guards a high-blast-radius path.
flaky, mostly failing	< 0.60	Block. Treat it as failing until proven otherwise.
fixtures changed	digest mismatch vs. baseline	Block pending an explicit re-baseline in the PR body.

The "mostly passing" row is where teams cheat. A case at 0.6 passes four times out of five, so it will merge bad code eventually. Quarantine is a debt ticket, not a fix.

Step 4: Two hooks, two different jobs

Run a cheap sample before you push, and a strict one in CI. The local hook should be fast enough that nobody disables it.

#!/usr/bin/env bash
# .githooks/pre-push
set -euo pipefail
SAMPLE_RUNS=3 FLAKY_BUDGET=0 python tools/sample_model_tests.py

Install it once per clone:

git config core.hooksPath .githooks
chmod +x .githooks/pre-push

The CI job runs more samples and uploads the ledger as an artifact, so a reviewer can read the evidence instead of trusting the checkmark.

model-flake-gate:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: pip install -r requirements.txt
    - name: Sample model-backed fixtures
      env:
        MODEL_BASE_URL: ${{ secrets.MODEL_BASE_URL }}
        MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
        SAMPLE_RUNS: "5"
        FLAKY_BUDGET: "0"
      run: python tools/sample_model_tests.py
    - uses: actions/upload-artifact@v4
      if: always()
      with:
        name: model-ledger
        path: artifacts/model-ledger.json

Pin action versions to commit SHAs in a real repository, and keep the API key out of logs — the sampler never needs to echo it.

Where MonkeyCode fits in this workflow

Sampling multiplies your run count by five, which is exactly the workload you do not want to pay per call for while you are still calibrating fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes MonkeyCode as an open-source project offering free model access — currently stated as 10 million free tokens — plus a free server option. For this workflow that combination is a reasonable place to run the sampler: point MODEL_BASE_URL at the free endpoint, and use the free server to host the sampler job so five-times runs do not eat your own CI minutes. Check the project's current terms yourself before you build a budget or a contract around them.

One caveat that is not a caveat but a requirement: temperature 0 reduces variance, it does not remove it. Ledgers are endpoint-scoped. If you switch providers or models, re-baseline, or you will compare two different distributions and call it a regression.

Who should not use this approach

If you need contractual latency or availability guarantees, data residency, or audited per-request logs, a free tier is the wrong foundation for your gate — use a provider you can hold to terms. If your model tests are a demo rather than a required check, the sampler is overhead; one run is fine. And if your fixtures assert on exact prose, fix the fixtures first, because sampling will only tell you what you already know.

The useful habit is smaller than the tooling: stop asking whether the test passed, and start asking how often. If you want to try the sampler before wiring a billing account, the free endpoint and free server are enough to get your first ledger written.

Comments

[Humam Moin]

amazing bro about testing AI outputs multiple times instead of trusting a single pass. I also like the idea of checking JSON structure and required fields rather than exact wording. That makes much more sense for LLM testing.