I keep seeing the same question in code reviews.
"Can we put this on the free tier?"
How do you actually know?
Most answers start with a curl and a stopwatch. That's the wrong tool.
A free model server is not a smaller machine. It's a shared queue with a soft promise. Confusing the two produces myths that survive for years.
Here are six I stopped believing. Plus a fit test that settles them.
Myth 1: A fast first byte means a fast server
TTFB measures queue position, not capacity.
You got lucky in the queue. The next request may wait ten times longer.
Measure sustained throughput, not the first chunk. Tokens per second tells you more. A full response beats a single latency number.
I learned this the hard way. My demo froze on stage.
Myth 2: More quota fixes performance
Quota is a ceiling. It does not remove variance.
Naive retry logic turns a bigger quota into more failures.
Design for the floor, not the ceiling. Exponential backoff with jitter beats a higher limit every time.
Myth 3: The free tier runs the same model, same config
Sometimes it does. Often it doesn't.
Providers may cap concurrency, change batching, or serve a quantized variant. The model card says one thing. The endpoint may do another.
Never assume parity. Probe the actual behavior.
Myth 4: If it works in curl, it works in production
Curl hides your client's sins.
No connection pooling. No timeout handling. No streaming buffer logic.
Your app may serialize requests, drop connections, or block on reads. The endpoint was fine. Your client was the bottleneck.
Myth 5: One run is a benchmark
One run is an anecdote.
Free tiers share capacity with strangers. Your neighbor's batch job changes your latency.
Run the same probe at least ten times, across different hours. Variance is the signal, not the average. Average hides the tail. The tail is what your users feel.
Myth 6: The provider is always the bottleneck
I blamed the provider for weeks once.
The real problem was my code. I opened a new connection per request. I read the full body before streaming.
Check your client before you blame the queue. Connection reuse alone often doubles throughput.
The corrected mental model
Free tier is a queue, not a machine.
You are borrowing shared capacity with a soft SLA. The question is not "is it fast?" The question is "does its variance fit my workload?"
That changes everything. You stop hunting for a faster endpoint. You start matching workloads to tiers.
The fit test
I wrote a small probe that classifies an endpoint by workload fit.
It runs repeated calls at three concurrency levels. It records TTFB, tokens per second, and error rate. Then it maps the results to a decision table.
# fitcheck.py — classify a free model endpoint by workload fit
# Assumes an OpenAI-compatible streaming endpoint. Adjust parsing for your provider.
import asyncio
import statistics
import time
import httpx
ENDPOINT = "https://your-free-endpoint.example/v1/chat/completions"
PROMPT = "Explain idempotency in 150 words."
ROUNDS = 10
CONCURRENCY_LEVELS = [1, 4, 8]
def estimate_tokens(text: str) -> int:
# Rough estimate: ~4 characters per token
return max(1, len(text) // 4)
async def one_call(client: httpx.AsyncClient) -> dict:
payload = {
"model": "your-model",
"messages": [{"role": "user", "content": PROMPT}],
"stream": True,
}
start = time.perf_counter()
try:
first_chunk = None
text = ""
async with client.stream("POST", ENDPOINT, json=payload) as response:
async for chunk in response.aiter_text():
if first_chunk is None:
first_chunk = time.perf_counter()
text += chunk
return {
"ok": True,
"ttfb": first_chunk - start if first_chunk else None,
"total": time.perf_counter() - start,
"tokens": estimate_tokens(text),
}
except Exception as exc: # noqa: BLE001
return {"ok": False, "error": str(exc)}
async def run_level(concurrency: int) -> list[dict]:
limits = httpx.Limits(max_connections=concurrency)
async with httpx.AsyncClient(limits=limits, timeout=60) as client:
tasks = [one_call(client) for _ in range(ROUNDS)]
return await asyncio.gather(*tasks)
def classify(results: list[dict]) -> dict:
ok = [r for r in results if r.get("ok")]
errors = [r for r in results if not r.get("ok")]
ttfb = [r["ttfb"] for r in ok if r.get("ttfb")]
tps = [
r["tokens"] / r["total"]
for r in ok
if r.get("total") and r["tokens"]
]
cv = statistics.pstdev(tps) / statistics.mean(tps) if len(tps) > 1 else 1.0
return {
"error_rate": len(errors) / len(results),
"median_ttfb": statistics.median(ttfb) if ttfb else None,
"median_tps": statistics.median(tps) if tps else None,
"throughput_cv": round(cv, 2),
}
async def main() -> None:
for level in CONCURRENCY_LEVELS:
results = await run_level(level)
print(f"concurrency={level}: {classify(results)}")
if __name__ == "__main__":
asyncio.run(main())
Run it at three different hours. Then map the rows to this table.
| Signal | Demo-fit | Batch-fit | Prod-fit |
|---|---|---|---|
| Median TTFB | < 5s | < 2s | < 1s |
| Throughput CV | < 1.0 | < 0.5 | < 0.2 |
| Error rate | < 20% | < 5% | < 1% |
What each signal means:
- Median TTFB tells you queue wait.
- Median TPS tells you throughput.
- CV tells you stability.
- Error rate tells you trust.
Example output (yours will differ):
concurrency=1: {'error_rate': 0.0, 'median_ttfb': 1.1, 'median_tps': 19.2, 'throughput_cv': 0.28}
concurrency=4: {'error_rate': 0.0, 'median_ttfb': 1.9, 'median_tps': 22.5, 'throughput_cv': 0.41}
concurrency=8: {'error_rate': 0.1, 'median_ttfb': 3.9, 'median_tps': 16.0, 'throughput_cv': 0.61}
Concurrency 1 and 4 pass batch-fit. Concurrency 8 drops to demo-fit. Verdict: batch-fit for low concurrency. Not prod-fit.
Read the table honestly. Demo-fit means demos. Batch-fit means offline jobs. Prod-fit means you have evidence, not hope.
Running it against MonkeyCode's free server
I used this exact script to evaluate MonkeyCode's free model access and free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The workflow is the same as for any provider. Point the script at the endpoint. Run it at three different hours. Compare the three rows.
I did not measure production-grade latency. That's not what the tier promises. I measured fit. That's the only honest question.
The script does not care which provider you point it at. That is the point.
The output tells you where the endpoint belongs. Demo. Batch. Or nowhere near your traffic.
Who should not use this approach
This fit test is not a substitute for an SLA.
If your workload is bursty, user-facing, or regulated, you need a contract, not a queue. Free tiers can change shape without notice. Your probe result is a snapshot, not a guarantee.
Also skip this if you cannot tolerate any error. The test measures variance. It does not remove it.
Skip this if you need data residency guarantees. A probe cannot verify where tokens are processed.
The takeaway
Stop asking if a free tier is fast.
Ask if it fits your workload. Then prove it with repeated measurements.
Run the fit test before your next architecture debate. The verdict settles the argument.
The queue is not the machine. Your job is to know which one you are using.
Top comments (0)