Gate a New Model on Reproducibility Before You Pay for Confidence

The first number that separates a usable model route from a demo is exact-match agreement across five identical calls with temperature 0. When that number is below 1.0, downstream pass rates are measuring sampling variance more than capability. Current model discussions tend to skip this check: teams point a new endpoint at an eval set, read one pass rate, and merge. That single number cannot tell you whether the route will produce the same shape twice.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access and free server option make the harness below easier to run without a local GPU, but the method is provider-agnostic. MonkeyCode's operator describes the current free access as 30,000,000 tokens plus a free server option; the exact allowance changes, so verify it before you commit CI minutes.

A full-stack model change needs three layers: a fixed task contract, a consumption budget, and a fail-closed threshold. The contract is a small JSONL file whose expected outputs are stable enough to assert. The budget is the maximum token spend allowed before you allow the route change. The threshold is repeated-run agreement, not a single best-of-n score. Best-of-n is an optimization procedure; it hides variance by pre-selecting the lucky answer. A deploy decision should be based on the distribution of outputs under the same contract.

Signal	Check	Starter fail-closed threshold
Exact agreement	Same output across five runs per case	< 1.0 per case
Reference pass rate	Output matches the expected normalized result	< 0.90
Token consumed	Summed usage across all runs	> configured budget
Latency tail	Maximum run latency per case	> 3x baseline median

The thresholds above are starting points, not universal rules. A high-risk route may need a higher pass rate, while an internal draft feature can tolerate more variance if it has no write path.

What a five-run gate actually measures

A five-run gate is a distribution check, not a quality benchmark. If a model returns three correct JSON objects and two different JSON keys from the same input, the failure is an integration failure: a downstream parser will see a contract violation. A single benchmark run may skip that case entirely. Running the same contract repeatedly makes the failure observable before it reaches user traffic.

Temperature 0 does not guarantee determinism across providers. Hardware, batching, and server-side changes can still affect output. This is why the gate treats any disagreement as a signal rather than waiting for a full benchmark to fail. The cost of a false negative is one blocked deployment. The cost of a false positive is a production route that sometimes returns a different contract.

The following script assumes an OpenAI-compatible chat completions endpoint. It reads cases.jsonl, runs each case five times, tracks token usage, and exits non-zero when the gate fails.

import json
import os
import statistics
import time
import httpx

BASE_URL = os.environ['BASE_URL']
MODEL = os.environ['MODEL']
API_KEY = os.environ.get('API_KEY', '')
RUNS = int(os.environ.get('RUNS', '5'))
TOKEN_BUDGET = int(os.environ.get('TOKEN_BUDGET', '500000'))

with open('cases.jsonl') as f:
    cases = [json.loads(line) for line in f]

passed = 0
used = 0
rows = []
for case in cases:
    answers = []
    latencies = []
    for _ in range(RUNS):
        started = time.perf_counter()
        response = httpx.post(
            f'{BASE_URL}/chat/completions',
            headers={'Authorization': f'Bearer {API_KEY}'},
            json={
                'model': MODEL,
                'temperature': 0,
                'messages': [
                    {'role': 'system', 'content': 'Return only the JSON object requested.'},
                    {'role': 'user', 'content': case['prompt']},
                ],
            },
            timeout=60,
        )
        response.raise_for_status()
        body = response.json()
        content = body['choices'][0]['message']['content'].strip()
        usage = body.get('usage') or {}
        used += usage.get('total_tokens') or (
            usage.get('prompt_tokens', 0) + usage.get('completion_tokens', 0)
        )
        answers.append(content)
        latencies.append(time.perf_counter() - started)

    expected = case['expected'].strip()
    exact = sum(1 for answer in answers if answer == expected)
    rows.append({
        'id': case.get('id'),
        'exact_agreement': exact / RUNS,
        'median_latency_ms': round(statistics.median(latencies) * 1000, 1),
        'max_latency_ms': round(max(latencies) * 1000, 1),
        'distinct_answers': sorted(set(answers)),
    })
    if exact == RUNS:
        passed += 1

report = {
    'tasks': len(cases),
    'passed': passed,
    'pass_rate': passed / len(cases),
    'token_used': used,
    'budget': TOKEN_BUDGET,
    'budget_exceeded': used > TOKEN_BUDGET,
    'rows': rows,
}
print(json.dumps(report, indent=2))

if passed != len(cases) or used > TOKEN_BUDGET:
    raise SystemExit(1)

The script is intentionally strict. For open-ended tasks, replace exact match with a normalized assertion or an embedding threshold and make the pass rate decimal rather than categorical. The important property is that the gate can fail without a human rereading every output.

Where the free server fits

You should run this gate before a new route becomes a dependency. If you already have a baseline endpoint, execute the script against it first to capture the baseline distribution. Then point BASE_URL at a free server option and use the free token allowance to evaluate the candidate without borrowing production quota. Keep the server read-only and do not attach it to user traffic.

Practice	Prompt playground	Acceptance gate
Goal	See if an answer feels correct	Reject unstable or out-of-budget routes
Input	One or two prompts	Stable JSONL contract
Evidence	A readable answer	Distribution over repeated runs
Decision	Eyeballing	Fixed pass/fail and exit code
Cost	Unbounded manual use	Predeclared token budget

The free server is most valuable when it runs the same contract that production would eventually exercise. If the harness points at a different prompt set, you are not measuring the route boundary; you are measuring a demo.

Limitations and who should skip this

• Exact match is wrong for high-entropy creative output. Use an assertion-based regression set instead.
• Five runs is a small sample. Increase RUNS or run the gate nightly if rare flake matters.
• Temperature 0 is not guaranteed determinism. Providers can still change outputs through batching or backend changes.
• A free tier and free server should not replace production capacity, retry policy, or observability.
• Skip this approach if the task has no stable reference contract; a variance gate needs a stable expected output.

Start with five stable cases, not fifty prompts. The gate becomes useful when it can fail closed before a new model reaches a route boundary. The free server option helps only if you feed it a fixed contract and a budget; otherwise you are still sampling.