I Sent One Prompt 50 Times. Here's the Repeatability Audit I Run on Free Servers.

One response looked perfect. Fast. Fluent. Free.

So I wired the endpoint into a CI job. Three days later, the job failed. Same prompt. Different output. No code change.

Was the model bad? Or was the server?

Most teams never separate those two questions. I built a 20-minute audit that does.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are exactly the kind of endpoint this audit targets.

Why repeatability matters

Free model servers sit behind shared queues. Rate limits. Variable backends. Paid endpoints hide most of that behind an SLA. Free ones don't.

One good response proves nothing. You need a distribution.

If you feed LLM output into tests, evals, or generated docs, variance is a bug. It turns green builds red. It turns golden tests into guesswork.

What I measure

Six signals. All cheap to collect.

1. Exact match rate — byte-identical outputs.
2. Similarity to the modal output — how close are the near-misses?
3. TTFT p50/p95 — time to first token.
4. Total latency p50/p95.
5. Error rate by status code.
6. Truncation rate — outputs that end early.

I skip quality scoring. This audit is about consistency, not cleverness.

The experiment design

Keep everything fixed. Change nothing but time.

• One prompt. Same wording every run.
• Temperature 0, if the server supports it.
• Fixed max_tokens.
• 50 runs. 2-second spacing.
• Same client. Same network. Same payload.

Record everything. Hash every output.

Why 50? Small enough to finish in 20 minutes. Large enough to expose rate limits and tail latency. If you need tighter confidence, run 100.

The script

Here's the full harness. Python 3.10+. One dependency: httpx.

# repeatability_audit.py
# Python 3.10+ | pip install httpx
import difflib
import hashlib
import json
import statistics
import time
from collections import Counter
from dataclasses import dataclass

import httpx

# --- config: adjust to your server ---
BASE_URL = 'https://your-server.example/v1/chat/completions'
MODEL = 'your-model'
API_KEY = 'anything'   # free servers often ignore this
PROMPT = 'Explain the difference between an index and a constraint in a database. Keep it under 80 words.'
N_RUNS = 50
SPACING_S = 2.0        # cooldown between requests
MAX_TOKENS = 256
TEMPERATURE = 0.0      # use 0 if the server supports it
# ------------------------------------

ENDINGS = {'.', '?', '!', ')', ']', chr(34), chr(39)}  # chr(34) = double quote, chr(39) = single quote


@dataclass
class Run:
    ok: bool
    status: int
    ttft_ms: float
    total_ms: float
    raw: str = ''
    text: str = ''
    error: str = ''


def extract_content(body: str) -> str:
    # Naive SSE extraction for OpenAI-style streams. Falls back to raw body.
    parts = []
    for line in body.splitlines():
        if line.startswith('data: '):
            data = line[6:]
            if data == '[DONE]':
                continue
            try:
                delta = json.loads(data)['choices'][0]['delta'].get('content', '')
                parts.append(delta)
            except Exception:
                continue
    return ''.join(parts)


def is_truncated(text: str) -> bool:
    # Heuristic: a complete answer usually ends with sentence punctuation.
    stripped = text.rstrip()
    return bool(stripped) and stripped[-1] not in ENDINGS


def run_once(client: httpx.Client) -> Run:
    payload = {
        'model': MODEL,
        'messages': [{'role': 'user', 'content': PROMPT}],
        'max_tokens': MAX_TOKENS,
        'temperature': TEMPERATURE,
        'stream': True,
    }
    headers = {'Authorization': f'Bearer {API_KEY}'}
    start = time.perf_counter()
    try:
        with client.stream(
            'POST', BASE_URL, headers=headers, json=payload, timeout=60.0
        ) as resp:
            next(resp.iter_bytes(), None)  # first chunk = TTFT marker
            ttft_ms = (time.perf_counter() - start) * 1000
            body = b''.join(resp.iter_bytes())
            total_ms = (time.perf_counter() - start) * 1000
            raw = body.decode('utf-8', errors='replace')
            text = extract_content(raw) or raw
            return Run(
                resp.status_code == 200,
                resp.status_code,
                ttft_ms,
                total_ms,
                raw=raw,
                text=text,
            )
    except Exception as exc:
        total_ms = (time.perf_counter() - start) * 1000
        return Run(False, 0, total_ms, total_ms, error=str(exc))


def percentile(sorted_values: list[float], p: float) -> float:
    if not sorted_values:
        return 0.0
    k = (len(sorted_values) - 1) * p / 100
    lo = int(k)
    hi = min(lo + 1, len(sorted_values) - 1)
    return sorted_values[lo] + (sorted_values[hi] - sorted_values[lo]) * (k - lo)


def similarity(a: str, b: str) -> float:
    return difflib.SequenceMatcher(None, a, b).ratio()


def main() -> None:
    runs: list[Run] = []
    with httpx.Client() as client:
        for i in range(N_RUNS):
            run = run_once(client)
            runs.append(run)
            print(
                f'run {i + 1:02d}/{N_RUNS}: '
                f'status={run.status} ttft={run.ttft_ms:.0f}ms '
                f'total={run.total_ms:.0f}ms'
            )
            time.sleep(SPACING_S)

    ok_runs = [r for r in runs if r.ok]
    outputs = [r.text for r in ok_runs if r.text.strip()]
    modal = Counter(outputs).most_common(1)[0][0] if outputs else ''
    exact = sum(1 for o in outputs if o == modal) / len(outputs) if outputs else 0.0
    sims = [similarity(o, modal) for o in outputs] if outputs else []
    ttfts = sorted(r.ttft_ms for r in ok_runs)
    totals = sorted(r.total_ms for r in ok_runs)

    print('\n=== repeatability report ===')
    print(f'runs={N_RUNS} ok={len(ok_runs)} errors={len(runs) - len(ok_runs)}')
    print(f'status codes: {dict(Counter(r.status for r in runs))}')
    print(f'exact match vs modal output: {exact:.1%}')
    print(f'median similarity: {statistics.median(sims):.3f}')
    print(f'unique outputs: {len(set(outputs))}/{len(outputs)}')
    print(f'truncated (heuristic): {sum(1 for o in outputs if is_truncated(o))}/{len(outputs)}')
    if ttfts:
        print(f'ttft p50={percentile(ttfts, 50):.0f}ms p95={percentile(ttfts, 95):.0f}ms')
    if totals:
        print(f'total p50={percentile(totals, 50):.0f}ms p95={percentile(totals, 95):.0f}ms')
    if modal:
        print(f'modal output hash: {hashlib.sha256(modal.encode()).hexdigest()[:12]}')


if __name__ == '__main__':
    main()

The script assumes an OpenAI-style chat completions endpoint. Adjust BASE_URL, MODEL, and the payload to match your server. Free servers often ignore API_KEY; pass anything.

How to run it

pip install httpx
python repeatability_audit.py

Run it from the same machine that calls the endpoint in production. Run it at the time of day you actually use the server. A server that passes at 9 AM can fail at 2 PM.

Reading the output

Here's a sample report. It shows the format. It is not a measurement of any server on any date.

=== repeatability report ===
runs=50 ok=48 errors=2
status codes: {200: 48, 429: 1, 503: 1}
exact match vs modal output: 64.6%
median similarity: 0.91
unique outputs: 18/48
truncated (heuristic): 3/48
ttft p50=412ms p95=1840ms
total p50=3.1s p95=11.4s
modal output hash: 3f9a2c1d77e0

Here's how I read it.

• 64.6% exact match. Mostly deterministic at temperature 0. Not fully. String-compare CI assertions are too risky.
• 18 unique outputs. Near-misses are close (0.91 similarity). Fine for drafts. Bad for golden tests.
• TTFT p95 is 4.5× the p50. Tail latency. A 5-second timeout will fail intermittently.
• 2 errors in 50. That's 4%. Yellow zone.
• 3 truncated outputs. The server sometimes stops mid-thought. Check the usage field for a token cap.

The modal output hash is a fingerprint. Compare it across weeks. If it changes, the model or the prompt template changed.

The decision table

Signal	Green	Yellow	Red
Exact match rate	≥ 80%	40–80%	< 40%
Error rate	< 2%	2–10%	> 10%
TTFT p95 / p50	< 3×	3–10×	> 10×
Unique outputs	≤ 5	6–20	> 20

Green means safe for evals and golden tests, with a tolerance window. Yellow means fine for chat and drafts. Add retries and timeouts. Red means keep it out of CI. Cache outputs or pay for an endpoint.

What the audit usually reveals

Four patterns show up again and again.

• Throttle curve. First 15 runs are clean. Then 429s and rising latency. You hit a rate limit. Raise SPACING_S or lower concurrency.
• Truncation ceiling. Many outputs stop at the same character count. The server caps max_tokens below what you asked. Check the usage field.
• Cold start. Run 1 is 10× slower than run 5. Keep a warm connection. Or accept the first-call penalty.
• Two clusters. Outputs split into groups with low cross-similarity. The server may route to different backends. That's a dealbreaker for evals.

Limitations

This is a snapshot, not an SLA. Free servers change behavior weekly. Re-run the audit before you trust a new config.

Your network path is part of the measurement. Run it from the same place your workload runs.

The audit measures server plus model together. To isolate the model, run the same prompt against a paid endpoint.

50 runs is a small sample. Use 100 or more if you need tighter confidence intervals.

The truncation check is a heuristic. It flags missing punctuation, not real token counts. Pair it with the usage field if your server reports one.

Who should skip this

Teams with a paid SLA. You don't need a repeatability audit.

Teams testing model quality. This won't tell you if the model is smart. It tells you if the endpoint is consistent.

Teams that can't tolerate any output variance. Don't audit. Add a content-addressed cache instead.

The takeaway

One response is an anecdote. Fifty responses are a signal.

Run this audit before you wire a free endpoint into anything. It takes 20 minutes. It will tell you more than the first token ever did.