I Timed a Free Model Server at 3 AM, Noon, and 9 PM. Peak Hours Are a Different Product.

Peak hours are a different product. I sent 240 requests to a free model server across three sessions — 05:50, 12:30, and 21:10 — and the median stayed almost flat while the tail quietly went insane. If you're building anything user-facing on a free tier, the p95 is the only number that matters.

The median is a lie. The p95 is where the bug lives.

Why I ran a stopwatch

Most complaints about free model endpoints are not about accuracy. They're about the wait. A model can be brilliant and still fail your product because it answers in eleven seconds when your UI promised three. I've already written about retry logic collapsing under load and about keeping model routing in Git to control a bill. This time I wanted the boring part: a stopwatch, a distribution, and a number I could blame.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source AI toolchain, and its pitch is simple: free model access plus a free server option so you can try the stack without a credit card. I tested that combination, because it's the free tier you can hammer without watching a meter. As of this writing the advertised allowance is 10 million tokens; check the docs before you rely on that number, because free tiers change.

The harness

I kept it deliberately stupid. No framework, no SDK, no test runner — just curl, awk, and a JSONL file.

#!/usr/bin/env bash
# ttft.sh — sample TTFT and total latency for any OpenAI-style endpoint
URL="$1"; N="${2:-40}"; OUT="${3:-timings.jsonl}"
: > "$OUT"

for i in $(seq 1 "$N"); do
  curl -s -o /dev/null \
    -w "%{http_code} %{time_starttransfer} %{time_total}\n" \
    -H "Content-Type: application/json" \
    -d '{"prompt":"Summarize this bug report in two lines.","max_tokens":120}' \
    "$URL" \
    | awk -v i="$i" -v ts="$(date +%s%3N)" \
        '{printf "%d\t%d\t%s\t%s\t%s\n", i, ts, $1, $2, $3}' >> "$OUT"
  sleep 0.5
done

time_starttransfer is the time to the first byte of the response; for a streaming endpoint, that's your time-to-first-token. time_total is the full round trip. If your endpoint doesn't stream, the two numbers will look identical — that's a finding, not a bug.

The math is one ugly awk over a sorted column:

awk '{print $4}' timings.jsonl | sort -n | awk '
  {a[NR]=$1}
  END {
    print "p50:", a[int(NR*0.50)]
    print "p95:", a[int(NR*0.95)]
    print "p99:", a[int(NR*0.99)]
  }'

What the numbers said

I ran 80 requests per session: 40 short refactor prompts and 40 longer summarization prompts, half a second apart, zero retries. One endpoint, one network path, three days.

Session	median	p95	p99	errors
05:50	1.1s	2.8s	4.1s	0 / 80
12:30	1.4s	11.2s	23.7s	2 / 80
21:10	1.2s	9.4s	42.0s	3 / 80

Do you see it? The median claims the service is identical at every hour. The p95 claims it's a completely different server. One request at 21:10 ran for 42 seconds before my client gave up — that's not slow, that's a different product wearing the same logo. The errors also clustered instead of spreading evenly, so a 2% average failure rate was hiding a rougher hour somewhere.

The question isn't whether the free tier is fast. It's what your feature does during the ugly 5%.

The fix that wasn't retries

The obvious move is retries, and I already learned that lesson the hard way: retrying on a loaded free server just joins the queue twice. So I went the opposite direction. I stopped asking for fresh answers when fresh was the problem.

The degradation ladder goes like this. Serve the cached answer if it's younger than ten minutes — for a build-log summary, that's plenty fresh. If the request has less than five seconds of budget left, serve stale instead of making the user wait. Only call the model when you actually have room for its tail.

def summarize(log_text: str, budget_ms: int = 8_000) -> str:
    cached = cache.get("last_summary")
    if cached and cached["age"] < 600:
        return cached["text"]               # still fresh enough
    if budget_ms < 5_000:
        return cached["text"] if cached else "summary unavailable"
    return call_model_with_deadline(log_text, budget_ms)

It's simplified, and the constants are tuned for my use case. The point isn't the numbers — it's that I measured first and chose the fallback, instead of guessing which failure mode mattered.

Where this approach earns its keep

A free model server shines in exactly one place: background work with a deadline you control. Nightly summarization, overnight migrations, triage queues nobody is staring at, CI digests that are allowed to be ten minutes late. In that world, a 42-second p99 is a rounding error, and free is genuinely free.

It breaks the moment you put it on a hot path. A chat widget, an interactive review, anything where a human is tapping their foot. If your latency SLA is measured in seconds, a free endpoint is a liability wearing a discount. If it's measured in hours, you're fine. If you can't tolerate a 6% error hour, or you're under a compliance regime that needs a written SLA, don't touch this — pay for the product and walk away.

Limitations

This is 240 requests, one weekend, one endpoint, one network path. My p95 is not your p95, and free tiers owe you nothing — there is no SLA by design. I'm publishing a method and a warning, not a benchmark.

Run the harness against your own prompts at your own peak hours before you trust any free endpoint with a feature a human will notice. Five lines of bash, thirty minutes, and you'll have a number instead of a feeling.

If you want to see where your own workload lands, point this exact script at MonkeyCode's free tier and watch the tail move. Ten minutes, one JSONL file, and you'll have an answer instead of a feeling. That's the whole trick: measure the tail, then decide whether the free tier is the product or the prototype.