Free model access is not a gift. It is a contract with someone else's rate limits, queueing policy, and maintenance schedule. Self-hosting inverts that contract: you own the latency, the GPU, and the 2 a.m. page. Most teams choose between the two by comparing price per token, and that is exactly how they end up with a production agent that stalls at 9:15 every morning.
Agent workloads are moving from demos to production, and the conversation has shifted from what models can do to what they cost to operate. The problem is that agent traffic does not look like chat traffic. A coding agent emits bursts of small requests — a tool call, a diff review, a short completion — separated by long idle gaps. That shape punishes endpoints optimized for steady throughput. A cost-per-token benchmark measures unit price, not whether the endpoint survives your burst pattern. The only honest test is to probe the endpoint the way your agent will actually call it.
Three questions decide the fit before any pricing math. First, what is your traffic shape: steady, bursty, or spiky? Second, what happens to your data when it crosses a third-party boundary? Third, how much operational slack do you have — can you babysit a self-hosted model, or does the endpoint need to be someone else's problem? Free hosted tiers win when the answers are steady, non-sensitive, and no-slack; self-hosting wins when they are spiky, sensitive, and you have the time.
Consider a concrete case. A background job that summarizes a few documents an hour is steady and forgiving; a free tier is almost certainly fine. An interactive coding agent that fires eight parallel tool calls while a developer waits is spiky and latency-sensitive; the same free tier can feel like a different product.
Here is a probe you can run against any OpenAI-compatible endpoint. It fires a fixed number of requests at a fixed concurrency, retries once after a 429, and reports success rate, rate-limit events, and latency percentiles. Run it twice: once at concurrency 1 for a baseline, once at the concurrency your agent actually uses. The difference between those two runs is the real cost of the endpoint.
'''probe_endpoint.py — fit test for a free or cheap model endpoint.
Usage:
export ENDPOINT_URL='https://...'
export ENDPOINT_KEY='your-key'
export PROBE_MODEL='model-name'
python probe_endpoint.py --requests 60 --concurrency 4
'''
import argparse
import asyncio
import os
import statistics
import time
import httpx
async def call_once(client, url, headers, payload):
t0 = time.perf_counter()
try:
r = await client.post(url, headers=headers, json=payload, timeout=60)
return r.status_code, (time.perf_counter() - t0) * 1000
except Exception as exc:
return 0, (time.perf_counter() - t0) * 1000
async def worker(client, url, headers, payload, sem, results):
async with sem:
status, ms = await call_once(client, url, headers, payload)
if status == 429:
# one recovery attempt: wait, then resend
await asyncio.sleep(2)
status, ms = await call_once(client, url, headers, payload)
results.append(('recovered', status, ms))
else:
results.append(('direct', status, ms))
async def main():
ap = argparse.ArgumentParser()
ap.add_argument('--requests', type=int, default=60)
ap.add_argument('--concurrency', type=int, default=4)
ap.add_argument('--prompt', default='Reply with the single word ok.')
args = ap.parse_args()
url = os.environ['ENDPOINT_URL']
key = os.environ['ENDPOINT_KEY']
model = os.environ['PROBE_MODEL']
headers = {'Authorization': f'Bearer {key}', 'Content-Type': 'application/json'}
payload = {
'model': model,
'messages': [{'role': 'user', 'content': args.prompt}],
'max_tokens': 8,
}
sem = asyncio.Semaphore(args.concurrency)
results = []
async with httpx.AsyncClient() as client:
tasks = [
worker(client, url, headers, payload, sem, results)
for _ in range(args.requests)
]
await asyncio.gather(*tasks)
ok = [ms for _, status, ms in results if status == 200]
limited = [ms for kind, _, ms in results if kind == 'recovered']
failed = [ms for _, status, ms in results if status not in (200, 429)]
print(f'requests={len(results)} ok={len(ok)} rate_limited={len(limited)} failed={len(failed)}')
if ok:
ok.sort()
p95 = ok[min(len(ok) - 1, int(len(ok) * 0.95))]
print(f'latency_ms p50={statistics.median(ok):.0f} p95={p95:.0f} max={ok[-1]:.0f}')
if limited:
print('429s observed; the probe waited 2s and retried once. If recovered entries are 429 again, the endpoint needs a longer cooldown than your agent timeout.')
if __name__ == '__main__':
asyncio.run(main())
Read the output like this. If p95 latency sits close to p50, the endpoint queues fairly under load. If p95 is three times p50, you are seeing contention, and your agent's timeouts will fire at the worst possible moment. If 429s appear at your expected burst size, the free tier's contract does not match your traffic shape. No retry logic fixes that; it only converts a rate limit into a token incinerator.
Record four numbers for each run: success rate, 429 count, p50, and p95. That is your endpoint's signature. Compare the signature at concurrency 1 and concurrency 8; if p95 doubles while success rate drops, the endpoint is not built for your agent's parallel tool calls. Some agents fan out ten tool calls at once, and a free tier that handles one request gracefully can still fail that pattern.
The probe results feed directly into the decision table below. A high 429 count at your working concurrency moves you to the "risky" row regardless of how cheap the tokens are.
| Workload shape | Free hosted tier | Self-hosted |
|---|---|---|
| Dev/test, low volume | Fits | Overkill |
| Steady production, non-sensitive | Fits with a fallback | Predictable but costly |
| Bursty, latency-sensitive | Risky | Better control |
| Regulated or private data | Avoid | Required |
| No operational slack | Fits | Do not attempt |
Notice what is missing from the table: price. Price decides which self-hosted option you pick, not whether you self-host. The free tier's real cost is coupling — your agent's availability inherits someone else's queue, and your p95 becomes their problem. A limitation like that is informative: it forces you to design backoff, fallbacks, and timeouts before you need them, which is exactly the discipline a production agent requires anyway.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option; at the time of writing, the free tier includes a 10M-token allowance. I have not benchmarked it here, and you should not trust a benchmark you cannot reproduce. What makes it relevant to this guide is that it is a legitimate candidate for the "free hosted tier" column — which means it deserves the same probe as any other endpoint. Point the script at its server, run the two concurrency passes, and record what you see.
Who should not use this approach? Teams with regulated data, hard latency ceilings, or predictably spiky traffic. Also anyone who treats a free tier as a permanent contract: free tiers change quotas, models, and terms without notice. Re-run the probe on a schedule, and configure a fallback endpoint before you need it, not after the first 429 in production.
The cheapest endpoint is the one that fails at the wrong moment. The probe costs twenty minutes and a few hundred tokens; the outage it prevents costs considerably more. Run it against MonkeyCode's free server if you want a concrete data point, but let the numbers — not the promise — make the decision.
Top comments (1)
The burst shape point is where these benchmarks usually get useful. I would add one more test before trusting a free endpoint with agent traffic. Run a short replay with the same tool-call gaps your agent produces, then score p95 latency and retry behavior, not just total token cost. A cheap endpoint that serializes the 9 a.m. burst can make a good agent feel broken.