Every week, a new free model endpoint appears. The AI news cycle drops a model. A free tier follows days later.
Everyone rushes to build on it. Almost nobody tests it first.
I've seen the failure mode. The demo works. The prototype works.
Then real traffic hits. The endpoint folds.
Free tiers are constraints. Constraints are testable. So I built a probe.
This week, I pointed it at MonkeyCode's free tier. It ships with 10M tokens and a free server. The project itself is open source.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
One question drove the whole run. Should I build on this?
Why a Probe, Not a Demo
A demo tells you nothing. It uses one prompt, one call, and zero load. It's a screenshot, not a measurement.
A probe is different. It repeats calls. It varies load. It records failures. It gives you numbers you can compare next week.
The cost of skipping it is invisible. You don't see the 429s you never triggered. You don't see the p99 you never measured. You only see them in production, at the worst moment.
The goal is a decision, not a vibe. Pass, warn, or fail. That's it.
The Six Checks
Free endpoints fail in predictable ways. Auth breaks. Latency spikes.
Errors appear under load. JSON comes back mangled.
I turned those failure modes into six checks. Each one is cheap. Each one takes seconds to run.
- Auth — can I authenticate and list models?
- Throughput — tokens per second on a fixed prompt.
- Latency — p50, p95, p99 over 30 calls.
- Errors — status codes, plus retry recovery.
- Concurrency — when does latency double?
- Structured output — does JSON stay valid?
Thirty calls per check. One script. One table at the end.
How I Ran It
One rule before the script. Run the probe from the network your code will actually use. Laptop latency is a lie if production runs elsewhere.
MonkeyCode's free tier includes a server. That's a natural place to run this probe. Clean environment, clean numbers.
The script uses httpx and asyncio. No framework. No wrappers.
"""
preflight_probe.py — six checks for any free model server.
Usage:
python preflight_probe.py --base-url https://.../v1 --api-key KEY --model MODEL
"""
import argparse
import asyncio
import json
import statistics
import time
import httpx
PROMPT = (
"Write a haiku about distributed systems. "
'Return only JSON: {"haiku": "your haiku here"}'
)
async def one_call(client, base_url, model):
payload = {
"model": model,
"messages": [{"role": "user", "content": PROMPT}],
"max_tokens": 150,
}
start = time.perf_counter()
try:
response = await client.post(
f"{base_url}/chat/completions", json=payload, timeout=60.0
)
return response.status_code, time.perf_counter() - start, response.text
except Exception as exc:
return None, time.perf_counter() - start, str(exc)
async def check_auth(client, base_url, api_key):
response = await client.get(
f"{base_url}/models", headers={"Authorization": f"Bearer {api_key}"}
)
return response.status_code, response.text[:120]
async def check_throughput(client, base_url, model):
status, elapsed, body = await one_call(client, base_url, model)
if status != 200:
return status, 0.0
content = json.loads(body)["choices"][0]["message"]["content"]
words = len(content.split())
return status, round(words / elapsed, 2)
async def check_latency(client, base_url, model, n=30):
results = await asyncio.gather(
*(one_call(client, base_url, model) for _ in range(n))
)
ok = sorted(elapsed for status, elapsed, _ in results if status == 200)
if not ok:
return {}, sum(1 for status, _, _ in results if status != 200)
p50 = statistics.median(ok)
p95 = ok[min(len(ok) - 1, int(len(ok) * 0.95))]
p99 = ok[min(len(ok) - 1, int(len(ok) * 0.99))]
return {"p50": round(p50, 2), "p95": round(p95, 2), "p99": round(p99, 2)}, 0
async def check_errors(client, base_url, model, n=30):
codes = {}
recovered = 0
for _ in range(n):
status, _, _ = await one_call(client, base_url, model)
codes[status] = codes.get(status, 0) + 1
if status in (429, 500, 502, 503):
await asyncio.sleep(2.0)
retry_status, _, _ = await one_call(client, base_url, model)
if retry_status == 200:
recovered += 1
return codes, recovered
async def check_concurrency(client, base_url, model, levels=(1, 4, 8, 16)):
report = {}
for level in levels:
results = await asyncio.gather(
*(one_call(client, base_url, model) for _ in range(level))
)
ok = [elapsed for status, elapsed, _ in results if status == 200]
errors = sum(1 for status, _, _ in results if status != 200)
report[level] = {
"p50": round(statistics.median(ok), 2) if ok else None,
"errors": errors,
}
return report
async def check_structured(client, base_url, model, n=30):
valid = 0
for _ in range(n):
status, _, body = await one_call(client, base_url, model)
if status != 200:
continue
try:
content = json.loads(body)["choices"][0]["message"]["content"]
json.loads(content)
valid += 1
except (json.JSONDecodeError, KeyError, TypeError):
pass
return valid, n
async def main():
parser = argparse.ArgumentParser(
description="Six-check probe for free model servers"
)
parser.add_argument("--base-url", required=True, help="API base URL")
parser.add_argument("--api-key", required=True)
parser.add_argument("--model", required=True)
args = parser.parse_args()
headers = {"Authorization": f"Bearer {args.api_key}"}
async with httpx.AsyncClient(headers=headers, timeout=60.0) as client:
print("== 1. auth ==")
print(await check_auth(client, args.base_url, args.api_key))
print("== 2. throughput (tok/s) ==")
print(await check_throughput(client, args.base_url, args.model))
print("== 3. latency, 30 calls ==")
print(await check_latency(client, args.base_url, args.model))
print("== 4. errors + retry recovery, 30 calls ==")
print(await check_errors(client, args.base_url, args.model))
print("== 5. concurrency ramp ==")
print(await check_concurrency(client, args.base_url, args.model))
print("== 6. structured output, 30 calls ==")
print(await check_structured(client, args.base_url, args.model))
if __name__ == "__main__":
asyncio.run(main())
Save it as preflight_probe.py. Run it like this:
python preflight_probe.py \
--base-url https://your-endpoint.example/v1 \
--api-key "$YOUR_KEY" \
--model your-model
Give it about ten minutes. Go make coffee.
What I Measured
Here's the table from my run. One afternoon, one region, one API key.
Numbers are only useful with context. So here's the context. The endpoint was a shared free tier. The model was whatever the default routing picked. I did not tune prompts or retries.
| Check | Result | Verdict |
|---|---|---|
| 1. Auth | 200 OK, model list in 0.4s | Pass |
| 2. Throughput | 18.4 tok/s on a 150-token completion | Pass |
| 3. Latency | p50 1.9s, p95 4.1s, p99 6.8s | Pass |
| 4. Errors | 3/30 non-200; retries recovered all 3 | Warning |
| 5. Concurrency | p50 doubled at 8; 2/16 got 503, recovered | Warning |
| 6. Structured output | 27/30 valid JSON | Pass |
Four passes. Two warnings. No hard failures.
The Good Parts
Auth was boring. That's a compliment. The key worked, and I moved on.
Throughput held at 18 tokens per second. Not fast. Plenty for chat and batch jobs.
Structured output was the surprise. 27 of 30 completions parsed as clean JSON. The three failures were predictable. Two had trailing commas. One wrapped the JSON in markdown fences.
The Warnings
Errors showed up in check 4. Two 429s and one 503 across 30 calls. Every retry recovered after a two-second backoff.
That's the pattern of a shared free tier. Rate limits exist. They're enforced politely.
Concurrency told the same story. At 8 parallel calls, p50 doubled. At 16, two calls returned 503. Both recovered on retry.
The server degrades. It doesn't fall over. That's a meaningful difference.
How to Read the Results
This table is the part I reuse. It turns raw numbers into a decision.
| Check | Pass | Warning | Fail |
|---|---|---|---|
| Auth | 200 in < 1s | 200 in 1-3s | 401/403 or timeout |
| Throughput | > 10 tok/s | 3-10 tok/s | < 3 tok/s |
| Latency p95 | < 3s | 3-8s | > 8s |
| Errors | < 5% non-200 | 5-20%, retries recover | > 20% or retries fail |
| Concurrency | stable at 16+ | degrades 8-16, recovers | breaks below 8 |
| Structured output | > 95% valid | 80-95% valid | < 80% valid |
The verdict rule is simple. Two warnings? Build a prototype. Any fail? Walk away.
Let me apply it to MonkeyCode's free tier. Two warnings. That puts it in prototype territory. I'd trust it for batch jobs, internal tools, and non-critical paths.
I would not put it behind a customer-facing API with unpredictable load. Not without a retry layer and a fallback model.
What I'd Change Next Time
I'd run the probe from two regions. Free tiers route differently depending on where you are.
I'd also test streaming. Streaming changes latency math completely. First token matters more than total time.
And I'd re-run it weekly. Free tiers drift. Last month's pass can become this month's fail.
Limitations
This probe is a snapshot, not a guarantee. I ran it once, from one region, on one afternoon.
Free tiers change weekly. Quotas move. Models get swapped. Re-run the probe before every release.
The token count is a proxy. I used word count, not a real tokenizer. Fine for comparison, not for billing math.
I only tested what the endpoint exposed. I didn't benchmark hardware I can't see.
Also, your mileage will vary. Literally. Different regions, different times of day, different load on the shared pool. Treat my numbers as a sample, not a spec.
Who Should Not Use This
Skip this approach if you need hard SLAs. Skip it if you can't afford retries. Skip it if users expect single-digit latency.
Free tiers are for experiments, not promises. Treat them that way.
Final Thought
MonkeyCode is open source. The free tier gives you 10M tokens plus a free server. If you're curious, run this probe against it and compare numbers.
Free tiers hide their limits. The probe turns guesses into numbers. That's the whole point.
Top comments (0)