Free Inference Has a Tail. I Probe It Before I Ship.

Everyone is arguing about whether agents are real. Meanwhile the thing that actually kills my runs is a queue. Free inference is not slow — its average is fine. Its worst 1% is where your run dies, and the average hides it perfectly.

So I stopped benchmarking endpoints by "does it answer nicely." I probe three numbers and one taxonomy before I let anything depend on a provider. The three numbers are time-to-first-token at p50, p90 and p99. The taxonomy is how it fails, because a 429 and a silent stall are not the same animal and should not share a retry policy.

Here is the whole harness. It is short on purpose — if you cannot read your probe in one sitting, you will not trust its output.

Start cheap: curl already tells you a lot

Before writing Python, spend twenty seconds on the blunt instrument. Ten identical requests, timed, no streaming:

for i in $(seq 1 10); do
  curl -sS -o /dev/null \
    -H "Authorization: Bearer $LLM_API_KEY" \
    -H 'Content-Type: application/json' \
    -w 'connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
    -d "{\"model\":\"$LLM_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ok\"}],\"max_tokens\":4}" \
    "$LLM_BASE_URL/chat/completions"
done

If connect is stable and ttfb swings by 10x across ten requests, you already know something. You are not paying for compute, you are paying for position in somebody's queue.

The probe: tail, not average

This runs N streaming requests at concurrency CONC, records time-to-first-token per request, and classifies every failure instead of averaging it away.

# probe_tail.py — measure a streaming endpoint's tail, not its average.
import asyncio, json, os, time
import httpx

BASE  = os.environ["LLM_BASE_URL"].rstrip("/")
KEY   = os.environ["LLM_API_KEY"]
MODEL = os.environ["LLM_MODEL"]
CONC  = int(os.environ.get("CONC", "8"))
N     = int(os.environ.get("N", "40"))
TTFT_BUDGET = float(os.environ.get("TTFT_BUDGET", "10"))  # seconds
PROMPT = "Reply with exactly one word: ok"

async def one(client, sem, idx, rows):
    async with sem:
        row = {"i": idx, "status": None, "ttft": None, "total": None,
               "chunks": 0, "chars": 0, "usage": None, "err": None}
        t0 = time.perf_counter()
        try:
            async with client.stream(
                "POST", f"{BASE}/chat/completions",
                headers={"Authorization": f"Bearer {KEY}"},
                json={"model": MODEL, "stream": True, "max_tokens": 8,
                      "stream_options": {"include_usage": True},
                      "messages": [{"role": "user", "content": PROMPT}]},
            ) as r:
                row["status"] = r.status_code
                if r.status_code != 200:
                    row["err"] = f"http_{r.status_code}"
                else:
                    async for line in r.aiter_lines():
                        if not line.startswith("data:"):
                            continue
                        data = line[5:].strip()
                        if data == "[DONE]":
                            break
                        obj = json.loads(data)
                        if obj.get("usage"):
                            row["usage"] = obj["usage"]
                        text = (obj.get("choices") or [{}])[0].get("delta", {}).get("content") or ""
                        if text and row["ttft"] is None:
                            row["ttft"] = time.perf_counter() - t0
                        elif row["ttft"] is None and time.perf_counter() - t0 > TTFT_BUDGET:
                            row["err"] = "ttft_over_budget"   # connected, but nothing came
                            break
                        row["chunks"] += 1
                        row["chars"] += len(text)
        except Exception as e:
            row["err"] = type(e).__name__
        row["total"] = time.perf_counter() - t0
        if row["ttft"] is None and row["err"] is None:
            row["err"] = "no_tokens"
        rows.append(row)

def pct(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(p / 100 * len(xs)))] if xs else None

async def main():
    sem, rows = asyncio.Semaphore(CONC), []
    limits = httpx.Limits(max_connections=CONC, max_keepalive_connections=CONC)
    async with httpx.AsyncClient(timeout=httpx.Timeout(60, connect=10), limits=limits) as c:
        t0 = time.perf_counter()
        await asyncio.gather(*(one(c, sem, i, rows) for i in range(N)))
        wall = time.perf_counter() - t0

    ok = [r for r in rows if r["err"] is None]
    ttft = [r["ttft"] for r in ok]
    report = {
        "n": N, "conc": CONC, "wall_s": round(wall, 2), "ok": len(ok),
        "ttft_p50": pct(ttft, 50), "ttft_p90": pct(ttft, 90), "ttft_p99": pct(ttft, 99),
        "tail_ratio": (pct(ttft, 99) / pct(ttft, 50)) if ttft and pct(ttft, 50) else None,
        "errors": {e: sum(1 for r in rows if r["err"] == e)
                   for e in {r["err"] for r in rows if r["err"]}},
    }
    print(json.dumps(report, indent=2))
    with open("probe_rows.jsonl", "w") as f:
        for r in rows:
            f.write(json.dumps(r) + "\n")

asyncio.run(main())

Run it at CONC=1, then CONC=4, then CONC=16. Same endpoint, same prompt, same minute. The shape of the report looks like this — and I want to be explicit that the block below is the structure, with placeholder values, not a measurement I am reporting:

{
  "n": 40, "conc": 8, "wall_s": 0.0, "ok": 0,
  "ttft_p50": null, "ttft_p90": null, "ttft_p99": null,
  "tail_ratio": null,
  "errors": {}
}

How to read it

tail_ratio is the number I actually act on. Under 3, the endpoint behaves like a service. Between 3 and 10, it behaves like a service with a bad neighbour. Above 10, you are not calling an API, you are entering a lottery, and every synchronous user-facing feature you build on it will feel broken one afternoon a week for reasons nobody can reproduce.

ttft_over_budget deserves its own paragraph, because it is the failure that fools people. The socket connects. The status is 200. Headers arrive. Then nothing, for fifteen seconds, until an intermediate proxy quietly drops you. Your average latency looks great because you never got a token to measure. That is why the probe records a missing TTFT as a failure instead of omitting the row. Silence is a result.

And no_tokens is the same lie in a different costume. A stream that opens and closes with zero content chunks returns valid-looking SSE and zero value. Counting it as success is how you end up with an agent that "worked" and produced an empty tool call three steps later.

Audit the token counter before you trust it

Free tiers are usually metered, so the meter is part of the evaluation. The usage block is the provider's number. Compute your own with the tokenizer for that model family, then compare per request:

# tokens_audit.py — compare provider-reported usage to a local count.
import json, tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # swap for your model's tokenizer
for line in open("probe_rows.jsonl"):
    r = json.loads(line)
    if not r["usage"]:
        continue
    local_prompt = len(enc.encode("Reply with exactly one word: ok"))
    drift = r["usage"]["prompt_tokens"] - local_prompt
    print(r["i"], "provider=", r["usage"]["prompt_tokens"], "local=", local_prompt, "drift=", drift)

A consistent positive drift is normal — chat templates add tokens you cannot see. An inconsistent drift, or a usage block that never appears on streamed responses, means your budget math is fiction. Plan from the larger of the two numbers, always.

Where MonkeyCode fits in this workflow

The reason I built this probe as a before-shipping habit rather than a one-off is that I needed somewhere cheap enough to run it repeatedly, at different concurrencies, on a schedule that does not depend on my laptop being awake. MonkeyCode is what I use for that: per the operator, the project offers free model access for the inference calls and a free server option, which is where I run the probe loop instead of from my desk. The operator states the free tier currently includes 10M tokens and a free server — treat that as a claim with a date on it and check the current terms before you plan a budget around them, because free-tier terms move and a probe harness that assumes yesterday's quota is a probe harness that lies to you.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I run the probe from the free server, keep the JSONL, and diff runs week over week. The harness is the artifact here — the endpoint is just what I point it at today. If you want to run the same thing, a free account is enough to get started; the value is in the diff, not the first run.

Decision table

Symptom in the report	Likely cause	What I do
p50 flat, tail ratio > 10	shared queue, bursty neighbours	ship only behind async jobs; never in a request path
429 at CONC=1	per-key rate limit, not load	budget per minute, not per run; back off with jitter
ttft_over_budget climbing with CONC	proxy idle timeout	cap concurrency below the knee; hard TTFT deadline in the client
no_tokens with status 200	dropped stream	treat as failure and re-run once with a deadline, not a loop
usage absent on streams	server ignores include_usage	count locally, bill locally, negotiate with numbers you own

Limitations, and who should not do this

This probe measures one endpoint from one network location at one moment. It is not a provider ranking, and running it once proves nothing — the diff between runs is the signal. It also measures first-token latency, not output quality; a fast wrong answer is still wrong.

If you have a hard p99 SLO on a user-facing path, if your workload needs contractual data residency, or if you cannot tolerate a free tier's terms changing mid-quarter, do not build on this. Pay for a provider with an SLA, or run your own box. Free inference is excellent for probes, batch jobs, internal tools, and the kind of evaluation work that gets better the more often you repeat it — and it is a poor foundation for anything you promised a customer.

So: before you argue about whether agents are real, go find out what your endpoint's worst 1% looks like. Would you ship on the number you have right now, or on the number you assumed?