Same Prompt, 50 Runs: A Variance Probe for Free Model Servers

One run tells you nothing.
Two runs tell you almost nothing.
Fifty runs start to tell the truth.

That's my rule for free model endpoints. A single response feels fast. A single answer looks correct. Then the next run is slow. Or different. Or both.

Was it the model? The server? The network? Your code?

Most people guess. I measure instead.

The Question Nobody Asks

Everyone benchmarks the model. Almost nobody benchmarks the endpoint's consistency.

A model can be brilliant. A server can still be chaotic. Free tiers add another layer of chaos. Queues, cold starts, shared capacity, rate limits.

The question is simple. How much does the same request vary?

If the answer is "a lot", your pipeline needs buffers. If the answer is "hardly at all", you can wire it straight into automation.

MonkeyCode offers free model access and a free server option. That's exactly the kind of endpoint this probe was built for.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Experiment Design

Keep it boring. That's the point.

• Same prompt. Every run.
• Same client. Same machine.
• No retries. No backoff.
• One request at a time.
• Temperature fixed at 0.
• 50 runs. One second pause between them.

I change nothing. I measure everything.

Three metrics matter most.

1. Latency variance. Total request time, run to run.
2. Output variance. How different are the answers?
3. Error rate. How often does the endpoint just fail?

Latency tells you about the server. Output tells you about the model. Errors tell you about the whole system.

Why 50 runs? Because 10 is a mood, not a measurement. With 50, the median stabilizes. The p95 stops jumping around. The outliers get a chance to show up.

The Probe

Here's the script. Plain Python. No frameworks. Standard library only.

#!/usr/bin/env python3
"""variance_probe.py — Run the same prompt N times against an LLM endpoint."""

import argparse
import json
import statistics
import time
from difflib import SequenceMatcher
from urllib import request, error


def call_endpoint(url, api_key, model, prompt, temperature):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
        "stream": False,
    }).encode("utf-8")

    req = request.Request(
        url,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {api_key}",
        },
        method="POST",
    )

    start = time.perf_counter()
    with request.urlopen(req, timeout=60) as resp:
        payload = json.load(resp)
    elapsed = time.perf_counter() - start

    content = payload["choices"][0]["message"]["content"]
    usage = payload.get("usage", {})
    return {
        "total_s": round(elapsed, 3),
        "output_chars": len(content),
        "prompt_tokens": usage.get("prompt_tokens", 0),
        "completion_tokens": usage.get("completion_tokens", 0),
        "content": content,
    }


def coefficient_of_variation(values):
    mean = statistics.mean(values)
    if mean == 0:
        return 0.0
    return statistics.pstdev(values) / mean


def percentile(sorted_values, p):
    if not sorted_values:
        return 0.0
    index = min(len(sorted_values) - 1, int(len(sorted_values) * p))
    return sorted_values[index]


def main():
    parser = argparse.ArgumentParser(description="Variance probe for LLM endpoints.")
    parser.add_argument("--url", required=True, help="Chat completions endpoint URL")
    parser.add_argument("--key", required=True, help="API key")
    parser.add_argument("--model", required=True, help="Model name")
    parser.add_argument("--prompt", default="Explain idempotency in one short paragraph.")
    parser.add_argument("--runs", type=int, default=50)
    parser.add_argument("--pause", type=float, default=1.0)
    parser.add_argument("--temperature", type=float, default=0.0)
    args = parser.parse_args()

    results = []
    errors = 0

    for i in range(1, args.runs + 1):
        try:
            result = call_endpoint(
                args.url, args.key, args.model, args.prompt, args.temperature
            )
            result["run"] = i
            results.append(result)
            print(
                f"run {i:02d}: {result['total_s']:6.2f}s  "
                f"{result['completion_tokens']:4d} tok",
                flush=True,
            )
        except (error.URLError, TimeoutError, KeyError, json.JSONDecodeError) as exc:
            errors += 1
            print(f"run {i:02d}: ERROR {type(exc).__name__}: {exc}", flush=True)
        time.sleep(args.pause)

    if not results:
        print("\nNo successful runs. Stop and check the endpoint.")
        return

    totals = [r["total_s"] for r in results]
    tokens = [r["completion_tokens"] for r in results]
    contents = [r["content"] for r in results]

    similarities = []
    for i in range(len(contents)):
        for j in range(i + 1, len(contents)):
            similarities.append(
                SequenceMatcher(None, contents[i], contents[j]).ratio()
            )

    sorted_totals = sorted(totals)
    print("\n=== REPORT ===")
    print(f"successful runs : {len(results)} / {args.runs}")
    print(f"error rate      : {errors / args.runs:.0%}")
    print(
        f"latency         : mean={statistics.mean(totals):.2f}s  "
        f"p50={percentile(sorted_totals, 0.50):.2f}s  "
        f"p95={percentile(sorted_totals, 0.95):.2f}s"
    )
    print(f"latency CV      : {coefficient_of_variation(totals):.2f}")
    print(f"token CV        : {coefficient_of_variation(tokens):.2f}")
    if similarities:
        print(
            f"output similarity: mean={statistics.mean(similarities):.2f}  "
            f"min={min(similarities):.2f}"
        )


if __name__ == "__main__":
    main()

The script expects a chat-completions style response. If your endpoint wraps responses differently, adjust call_endpoint. Then rerun.

Run it like this:

python variance_probe.py \
  --url "$MONKEYCODE_ENDPOINT" \
  --key "$MONKEYCODE_KEY" \
  --model "model-name-from-your-dashboard" \
  --prompt "Explain idempotency in one short paragraph." \
  --runs 50

Replace the variables with values from your MonkeyCode dashboard. The script prints one line per run. Then it prints a report.

The report has four numbers.

• Latency CV. Standard deviation divided by the mean.
• p95 latency. The slowest 5% of your runs.
• Token CV. How much the output length wobbles.
• Pairwise similarity. How similar every output is to every other output.

CV is the star. It normalizes variance. A CV of 0.1 means tight. A CV of 0.8 means chaos.

How to Read the Report

Don't look at one number. Look at the pattern.

Signal	Stable	Noisy	What it means
Latency CV	< 0.15	> 0.40	Queue or cold-start problems
Token CV	< 0.10	> 0.30	Unpredictable generation or truncation
Output similarity	> 0.90	< 0.70	Answers drift; avoid deterministic tasks
Error rate	0%	> 5%	Keep it out of synchronous paths

Three mistakes to avoid.

1. Running fewer than 20 runs. The p95 is meaningless with a tiny sample.
2. Changing the prompt mid-test. Then you're measuring two things at once.
3. Retrying failed runs manually. That hides the real error rate.

Keep the test boring. Boring tests give clean signals.

A Worked Example (Synthetic)

These numbers are synthetic. Run the probe to get real ones. This is how you read the output.

Scenario A: Stable

latency  mean=2.10s  p50=2.05s  p95=2.40s
latency  CV=0.08
tokens   mean=84.0  CV=0.06
output   similarity mean=0.94  min=0.88

This endpoint is boring. That's a compliment. The CV is low. The outputs barely drift.

Use it for batch jobs. Use it for code generation. Wire it into CI without fear.

Scenario B: Chaotic

latency  mean=4.80s  p50=3.10s  p95=11.90s
latency  CV=0.62
tokens   mean=71.0  CV=0.41
output   similarity mean=0.55  min=0.31

Same prompt. Same endpoint. Different universe every time.

The p95 is four times the median. That's a queue. The outputs barely agree. That's drift.

Don't put this in a user-facing path. Don't use it for tests. Cache aggressively. Or find another endpoint.

Why Variance Beats Speed

Fast endpoints lie. A single fast run hides a chaotic queue.

Variance reveals the real cost. Every slow run is a tax on your pipeline. Every retry doubles the tax.

Think about it. Which endpoint is better?

• Endpoint A. Mean 1.5s. p95 12s.
• Endpoint B. Mean 2.5s. p95 3.1s.

Endpoint A feels faster. Endpoint B is more predictable. For a synchronous call, B wins. For a batch job, A might win.

That's the decision variance enables. Speed alone can't make it.

What to Fix First

The probe separates three failure layers.

• Server noise shows up in latency CV.
• Model drift shows up in similarity.
• System fragility shows up in errors.

Each layer needs a different fix.

• High latency CV → add timeouts and a retry budget.
• Low similarity → cache outputs, pin prompts, avoid assertions.
• High error rate → move the call behind a queue or a worker.

Fix the worst layer first. Then rerun the probe. Compare the reports.

One probe run is a snapshot. Free servers change with the clock. Run it at different hours. Run it on different days.

Limitations

This probe is honest about its limits.

• It measures one client. One machine. One network path.
• It measures one time window. Not a week.
• It uses temperature 0. Real workloads vary temperature.
• It uses one prompt. Long prompts behave differently.
• Free tiers change. Today's numbers expire.

Treat the report as evidence, not prophecy.

Who Should Not Use This

Skip this probe if you need hard guarantees.

• Production user-facing latency. Not this.
• Regulated output. Not this.
• Contractual SLAs. Definitely not this.

Free endpoints are for experiments, prototypes, and tolerant workloads. The probe tells you how tolerant you need to be.

The Takeaway

One run is a coin flip. Fifty runs are a signal.

Run the probe before you trust an endpoint. Then run it again next week.

If someone says a free server is "fast enough", ask for the CV. Then ask for the similarity score. Then run the probe yourself.