Don't Trust One Run: A Time-of-Day Variance Probe for Free Model Servers

One run said the free server was fast.
The next run said it was broken.
Same prompt. Same client. Same payload.

Which one was true?

Neither. A single sample is not a measurement.
Free servers are shared. Routing changes. Neighbors appear and disappear.
Your one run just caught a random slice of someone else's day.

I wanted to know how much the endpoint changes across 24 hours.
So I built a small probe.
It sends the same 20 requests every hour and records latency percentiles plus error rates.

This article is the probe, the schedule, and the decisions I make from its output.
I won't quote my exact numbers here — they'd be stale by the time you read this.
Run the probe against your endpoint and get today's truth.

Why variance matters more than speed

Average latency is a lie.
A server can look fast in the median and still ruin your pipeline.

Three things break when an endpoint is inconsistent:

• Timeouts. You set them from one slow run. Now every run looks slow.
• Retries. You retry on a spike. The spike is that hour's normal state.
• Batch jobs. You schedule at midnight. Midnight is peak load somewhere else.

The fix is not a bigger timeout.
The fix is knowing which hours are stable.

The probe

I pointed the probe at the free model server that MonkeyCode offers.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The design is deliberately boring:

• One fixed prompt. Same length every run.
• Sequential requests. No concurrency noise.
• 20 requests per batch. Enough for percentiles, polite enough for a shared server.
• One batch per hour. Cheap to run all day.

Sequential matters.
If I fired 20 parallel requests, I'd measure my client's concurrency, not the server's state.
Boring is the point.

The script

Save this as probe.py.

import asyncio
import json
import os
import statistics
import time

import httpx

PROMPT = (
    "Explain why idempotency keys matter in distributed systems. "
    "Keep it under 80 words."
)
N = 20  # requests per batch
GAP = 0.2  # seconds between requests


async def probe_once(client, url, api_key):
    start = time.perf_counter()
    try:
        resp = await client.post(
            url,
            headers={"Authorization": f"Bearer {api_key}"},
            json={
                "messages": [{"role": "user", "content": PROMPT}],
                "max_tokens": 120,
            },
            timeout=60.0,
        )
        return {
            "ok": resp.status_code == 200,
            "status": resp.status_code,
            "latency": round(time.perf_counter() - start, 3),
        }
    except httpx.TimeoutException:
        return {"ok": False, "status": "timeout", "latency": None}
    except httpx.HTTPError as exc:
        return {"ok": False, "status": type(exc).__name__, "latency": None}


async def run_batch(url, api_key):
    async with httpx.AsyncClient() as client:
        results = []
        for _ in range(N):
            results.append(await probe_once(client, url, api_key))
            await asyncio.sleep(GAP)

    ok = sorted(r["latency"] for r in results if r["ok"])
    failed = [r for r in results if not r["ok"]]

    def pct(sorted_list, p):
        if not sorted_list:
            return None
        idx = min(len(sorted_list) - 1, int(len(sorted_list) * p))
        return sorted_list[idx]

    return {
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
        "total": N,
        "ok": len(ok),
        "error_rate": round(len(failed) / N, 3),
        "p50": pct(ok, 0.50),
        "p90": pct(ok, 0.90),
        "p95": pct(ok, 0.95),
        "max": ok[-1] if ok else None,
        "errors": [f["status"] for f in failed[:3]],
    }


if __name__ == "__main__":
    url = os.environ["ENDPOINT_URL"]
    api_key = os.environ.get("API_KEY", "")
    print(json.dumps(asyncio.run(run_batch(url, api_key)), indent=2))

Two notes before you run it:

• The model field is missing on purpose. Some endpoints require it, some reject it. Check your server's docs.
• max_tokens keeps responses short. You're measuring the server, not the model's essay skills.

Schedule it

A loop is not a schedule.
Cron works fine:

0 * * * * cd ~/variance-probe && \
  ENDPOINT_URL="$MC_FREE_URL" API_KEY="$MC_FREE_KEY" \
  python3 probe.py >> variance.jsonl

Prefer GitLab CI? Same idea, with artifacts:

# .gitlab-ci.yml
variance-probe:
  script:
    - python3 probe.py >> variance.jsonl
  artifacts:
    paths:
      - variance.jsonl
    expire_in: 30 days
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

Each line is one batch.
After 24 hours you have 24 data points.

Summarize with jq:

jq -r '[.timestamp, .p50, .p95, .error_rate] | @tsv' variance.jsonl

Example output:

2026-08-21T09:00:00  1.8   3.1   0.000
2026-08-21T14:00:00  4.2   11.7  0.100
2026-08-21T19:00:00  2.9   6.4   0.050

That is the shape of a shared server.
Morning was calm. Afternoon was noisy. Evening sat in between.
Your hours will differ. Your shape will not.

The decision table

Raw numbers are not decisions.
Here is how I turn the output into action.

Condition	Action
p95 < 2x p50, error_rate < 2%	Stable window. Run batch jobs here.
p95 2-4x p50, error_rate 2-10%	Degraded. Raise timeout, disable retries.
p95 > 4x p50 or error_rate > 10%	Bad window. Serve from cache or fall back.

Three rules I actually follow:

1. Schedule heavy work in stable windows. The probe tells you which ones they are.
2. Never set timeouts from one run. Use the p95 of your best stable window.
3. Retries are a last resort. If the hour is bad, retrying just adds load.

Limitations

This probe has edges. Know them before you trust it.

• It measures one endpoint. Your network path is part of the result.
• It uses one prompt. Long prompts may behave differently. Test your real payload size.
• It can't see the server's internals. You get symptoms, not causes.
• It's a snapshot of a shared system. The shape shifts as users come and go.

Who should not use this?

• Teams with hard SLAs. A free server is not your only path. It shouldn't be.
• Low-volume apps. If you make 50 calls a day, caching beats probing.
• Latency-critical features. If a user waits on every call, a variance probe won't save you.

The takeaway

One run tells you nothing.
Twenty runs tell you a little.
A day of runs tells you when to call.

Run the probe once. Save the JSON.
The next time the endpoint feels slow, you'll know whether it's you or the hour.