Fail the Eval When Samples Disagree

A golden case that flips across identical samples is already a regression, even when mean pass rate looks fine. Teams that store only a boolean per prompt treat that flip as noise, which is how flaky model behavior ships. The right unit of failure is disagreement among draws, not a single lucky completion that happened to match. Treat that pattern like a flaky unit test that passed on the retry you happened to watch in CI.

Traditional eval dashboards copy the green-or-red instinct of a compiler, then average those flags into a weekly chart. That average is a weak instrument whenever temperature is not zero, because two calls with the same prompt are not the same trial. A model that answers correctly four times and invents a field on the fifth has a stability defect, not a rounding error. The load-balancer analogy is useful here: one dying replica in five leaves p50 healthy while the tail burns.

n-sample evaluation is a volume problem before it is a modeling problem, which is why cheap endpoints change the method. MonkeyCode is an open-source project with free model access and a free server option, which makes repeated draws cheap enough to measure. Disclosure: This article was prepared as part of MonkeyCode's product outreach, and the claims above stop at those two availability options. The harness talks to a generic base URL and remains useful if that product is never used.

The workflow below scores three axes on every golden case: inter-sample agreement, JSON schema validity, and a completion-length budget. Agreement is the primary gate because it fails the case as soon as draws stop matching each other under a chosen grader. Schema and length are secondary gates that catch contract drift and cost growth when a human still calls the text close enough. A small Python script, a JSONL golden file, and an OpenAI-compatible chat endpoint are sufficient to reproduce the check without a vendor lock-in.

The following files are a proposed, unexecuted example rather than a measured production result from a live deployment. Replace the base URL, model identifier, and API key from your own environment; this article does not name a specific hosted model. Keep temperature above zero on purpose, because the harness is designed to observe spread rather than to hide it.

{"id":"invoice_extract_v3","prompt":"Extract vendor, total_cents, and currency as JSON. Invoice text: Acme Tools — total $19.50 USD.","expect_keys":["vendor","total_cents","currency"],"max_completion_tokens":80,"n":5,"agreement_min":0.8}
{"id":"refund_abstain_v1","prompt":"Return JSON with status=unknown when the refund policy is not in the prompt. User: can I refund order 9?","expect_keys":["status"],"max_completion_tokens":40,"n":5,"agreement_min":1.0}

Each golden line is a contract for the grader, not a loose vibes check against whatever text looks plausible. The first case demands key stability across five draws and a tight length budget so a rambling preface cannot sneak through. The second case is stricter: every sample must agree, because an abstention policy that only holds on some draws is not a policy. Golden files like this should sit beside the prompt template so a pull request shows the instruction change and the new spread gate.

#!/usr/bin/env python3
"""Proposed n-sample eval harness. Label: unexecuted example."""
from __future__ import annotations

import hashlib
import json
import os
import sys
import urllib.request
from collections import Counter
from pathlib import Path

ENDPOINT = os.environ.get("EVAL_BASE_URL", "http://127.0.0.1:8080/v1/chat/completions")
MODEL = os.environ.get("EVAL_MODEL", "local-model")
API_KEY = os.environ.get("EVAL_API_KEY", "")


def chat(prompt: str, max_tokens: int) -> str:
    payload = json.dumps(
        {
            "model": MODEL,
            "temperature": 0.7,
            "max_tokens": max_tokens,
            "messages": [{"role": "user", "content": prompt}],
        }
    ).encode()
    req = urllib.request.Request(
        ENDPOINT,
        data=payload,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {API_KEY}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]


def parse_obj(text: str) -> dict | None:
    text = text.strip()
    try:
        start, end = text.index("{"), text.rindex("}") + 1
        obj = json.loads(text[start:end])
    except (ValueError, json.JSONDecodeError):
        return None
    return obj if isinstance(obj, dict) else None


def canonical(obj: dict, keys: list[str]) -> str:
    clipped = {k: obj.get(k) for k in keys}
    return json.dumps(clipped, sort_keys=True, default=str)


def grade_case(case: dict) -> dict:
    keys = case["expect_keys"]
    draws = []
    for _ in range(int(case["n"])):
        raw = chat(case["prompt"], int(case["max_completion_tokens"]))
        obj = parse_obj(raw)
        draws.append(
            {
                "raw": raw,
                "ok_schema": obj is not None and all(k in obj for k in keys),
                "canon": canonical(obj, keys) if obj else None,
                "chars": len(raw),
            }
        )
    schema_pass = sum(d["ok_schema"] for d in draws) / len(draws)
    canons = [d["canon"] for d in draws if d["canon"]]
    top = Counter(canons).most_common(1)
    agreement = (top[0][1] / len(draws)) if top else 0.0
    mean_chars = sum(d["chars"] for d in draws) / len(draws)
    length_pass = mean_chars <= int(case["max_completion_tokens"]) * 4
    passed = (
        agreement >= float(case["agreement_min"])
        and schema_pass == 1.0
        and length_pass
    )
    return {
        "id": case["id"],
        "passed": passed,
        "agreement": round(agreement, 3),
        "schema_pass": round(schema_pass, 3),
        "mean_chars": round(mean_chars, 1),
        "fingerprint": hashlib.sha256(case["prompt"].encode()).hexdigest()[:12],
        "modes": top[:2],
    }


def main() -> int:
    path = Path(sys.argv[1] if len(sys.argv) > 1 else "golden.jsonl")
    results = [
        grade_case(json.loads(line))
        for line in path.read_text().splitlines()
        if line.strip()
    ]
    print(json.dumps(results, indent=2))
    failed = [r for r in results if not r["passed"]]
    print(f"failed {len(failed)} / {len(results)}", file=sys.stderr)
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it as a CI step that may be slow, because five draws per case is the measurement rather than a tax on the pipeline. The commands below assume an OpenAI-compatible server is already listening, which may be a local process or a remote free server you control. Fail the job on a non-zero exit so disagreement cannot hide inside a yellow dashboard tile that nobody opens.

export EVAL_BASE_URL="http://127.0.0.1:8080/v1/chat/completions"
export EVAL_MODEL="local-model"
export EVAL_API_KEY="replace-me"
python3 harness.py golden.jsonl
echo $?  # non-zero means disagreement, schema breakage, or length overrun

Read the JSON result as a lab notebook, not as a trophy that sits on a team wiki until the next model swap. The fingerprint field ties the score to the prompt bytes so a silent template edit cannot hide behind the same case id. The modes field shows the competing canonical answers, which is the closest this harness gets to a stack trace for a non-deterministic failure. If agreement falls while schema_pass stays at 1.0, the model is producing valid JSON that means different things on different draws.

A worked numeric example makes the gate less abstract, because a dashboard color hides the actual split among draws. Suppose five draws for invoice_extract_v3 produce four copies of one canonical object and one copy that stores 19.50 in total_cents. Agreement is 0.8, which meets a threshold of 0.8 only if the team accepts that unit bug on one fifth of traffic. Lowering agreement_min to 0.79 would green the case, which is how averages get gamed; keep the threshold honest or delete the case.

For refund_abstain_v1, a single draw that answers status=eligible should fail the whole row, because the policy did not hold. Abstention is a safety property, and safety properties do not get a quorum discount just because other samples looked careful. Store the offending raw completion beside the score so the next prompt edit has a concrete counterexample instead of a feeling. That counterexample is more useful in review than a pass-rate chart that moved by two points and then reversed.

This method will lie if every sample agrees on the wrong business result, because agreement is not the same thing as truth. Pair it with a frozen expected value for keys that have an objective answer, or the harness becomes a consensus engine for shared hallucinations. Length budgets also mislead when the task requires citations or multi-step tool traces, so raise the budget with the case rather than globally. Schema checks do nothing for free-form prose, so do not force JSON onto a writing task just to make the grader easy.

Operators should skip this approach when completions are already deterministic at temperature zero and the output is a closed enum with no tail risk. They should also skip it when a human reviewer must see every answer for regulatory reasons, since an agreement score cannot replace that review. High-stakes medical, legal, or financial generation needs a labeled gold answer and a person in the loop, not this script alone. If sampling five times still exceeds the latency budget, cut the suite to historically flaky cases rather than silently dropping n back to one.

The practical habit is to fail the build on disagreement first, and only then argue about which mode is correct. That ordering keeps prompt edits from shipping under the excuse that the eval is noisy when the model is the noisy part. Repeatable draws are what make that ordering possible, which is why n must stay greater than one after the suite feels stable. If an OpenAI-compatible endpoint is already running, execute the script against it and keep the cases whose modes nobody can explain.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.