Free Server Myth FAQ: What Your Queue Probe Tells You

You send a prompt.
Nothing.
Four seconds later, the answer arrives.

"Slow model," you say.

Stop.
The model probably wasn't slow.
Your request was standing in a queue.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option.
That combination is a convenient lab for the experiment below.
But the probe works with any OpenAI-compatible endpoint.

The myth inventory

Developers repeat the same lines when a free endpoint feels slow.

• "The free server is a tiny computer just for me."
• "Slow response means the model is bad."
• "Retrying faster will fix it."
• "Paid plans are the only way to ship."

Every one of those can be checked with one small artifact.

The 10-minute queue probe

The probe measures two things:

• ttft — time to first token
• total — full request time

ttft includes queue wait plus network.
total adds the rest of the generation.

# probe.py — queue.wait vs generation probe.
# Works with any OpenAI-compatible endpoint.
import argparse
import asyncio
import statistics
import time

from openai import AsyncOpenAI

MODEL = "replace-with-model"
BASE_URL = "replace-with-endpoint"  # e.g. https://your-server.example.com/v1
API_KEY = "replace-with-key"

client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)


async def one(call_id: int):
    sent = time.perf_counter()
    try:
        stream = await client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": "Reply with one word: ready"}],
            max_tokens=10,
            stream=True,
        )
        first = None
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                first = time.perf_counter()
                break
        total_ms = (time.perf_counter() - sent) * 1000
        ttft_ms = (first - sent) * 1000 if first else None
        return {"ok": True, "total_ms": total_ms, "ttft_ms": ttft_ms}
    except Exception as exc:
        return {"ok": False, "total_ms": (time.perf_counter() - sent) * 1000, "error": str(exc)}


async def run(concurrency: int, count: int):
    sem = asyncio.Semaphore(concurrency)

    async def worker(i: int):
        async with sem:
            return await one(i)

    return await asyncio.gather(*[worker(i) for i in range(count)])


def show(label: str, values: list[float]):
    if not values:
        print(f"{label}: no successful calls")
        return
    p50 = statistics.median(values)
    p95 = sorted(values)[max(0, int(len(values) * 0.95) - 1)]
    print(f"{label}: p50={p50:.0f}ms p95={p95:.0f}ms")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--concurrency", type=int, default=5)
    parser.add_argument("--count", type=int, default=20)
    args = parser.parse_args()

    results = asyncio.run(run(args.concurrency, args.count))
    ok = [r for r in results if r["ok"]]
    ttft = [r["ttft_ms"] for r in ok if r["ttft_ms"]]
    total = [r["total_ms"] for r in ok]

    show("time-to-first-token", ttft)
    show("total-request", total)
    print(f"ok={len(ok)}/{len(results)}")

Run it twice.

python probe.py --concurrency 1 --count 10
python probe.py --concurrency 10 --count 50

What to compare

Look at the numbers, not the vibes.

• If ttft rises with concurrency, your requests are queueing before the first token.
• If ttft stays flat but total rises, the delay is in generation, not the queue.
• If ok/total collapses at higher concurrency, your client logic is the problem.

The last one is gold.
A tight timeout plus aggressive retries turns one slow request into ten.
That is not a server bug.
That is a thundering herd you created.

The FAQ

Q: A free server is a small machine just for me?

No.
It is a shared queue.
Same elevator, same coffee line.
Your request waits until a seat is free.

Q: Why do all my requests feel slow at the same time?

Because the queue is global.
One noisy neighbor affects everyone.
That is not a conspiracy.
That is a shared resource.

Q: Should I retry faster?

No.
More retries mean more entries in the same queue.
Use exponential backoff, not panic loops.

Q: Does slow mean the model is worse?

No.
Latency is not quality.
This probe only measures delivery time.
A slow answer can be excellent.
A fast answer can be wrong.

Q: What matters more: free server or free model?

For latency, the server matters at low concurrency.
For output quality, the model matters.
Do not tune a model to fix a queue.

The corrected mental model

A free server is a seat in a shared coffee shop.
Free model access is a shared ordering counter.
You can get work done.
You cannot reserve the table.

Your code should assume the queue exists.
Set sane timeouts.
Add backoff.
Measure before you blame.

Limitations

This probe measures latency only.
It says nothing about safety, correctness, or cost.
It cannot split network time from server queue time.
Run it from two locations if you need a cleaner signal.

Who should skip this

• Teams with a hard p99 latency contract
• Model-evaluation teams writing quality reports
• Anyone with strict data residency requirements

Those teams need dedicated capacity or a formal benchmark harness.

One last thing

Run the probe once and keep the output.
The next time a model feels slow, you will have numbers.
Most of the time, those numbers will point at your queue.

The model is probably fine.
The queue is just crowded.