Free capacity has no price, but it does have a shape. That shape bends at some concurrency level you have probably never measured, and everything above the bend is queue time you paid for with tokens you thought you saved.
The usual math is volume math. You multiply your monthly tokens by zero, compare it to a paid invoice, and declare a winner. The problem is that the comparison only holds while requests arrive one at a time. Put sixteen of them in flight and the free path stops being a discount and starts being a latency budget you are spending down.
I keep a small harness for exactly this question. It ramps concurrency against any OpenAI-compatible endpoint and prints one row per level: latency percentiles, retry rate, and an effective cost per thousand completions that charges you for waiting as well as for tokens. You run it, you get your numbers. Nobody can hand you these numbers from the outside.
# sweep.py — run with: BASE_URL=... API_KEY=... MODEL=... python sweep.py
import asyncio, os, statistics, time
import httpx
BASE = os.environ["BASE_URL"].rstrip("/")
KEY = os.environ["API_KEY"]
MODEL = os.environ["MODEL"]
PROMPT = os.environ.get("PROMPT", "Reply with one sentence about queueing.")
MAX_TOK = int(os.environ.get("MAX_TOK", "64"))
LEVELS = [1, 2, 4, 8, 16, 32]
PER_LEVEL = 64
HOURLY_RATE = float(os.environ.get("HOURLY_RATE", "60")) # loaded cost of an engineer-hour
PRICE_IN = float(os.environ.get("PRICE_IN", "0")) # per 1M tokens, pa
PRICE_OUT = float(os.environ.get("PRICE_OUT", "0")) # per 1M tokens, paid path
async def one(client, sem):
async with sem:
attempts, t0 = 0, time.perf_counter()
while True:
attempts += 1
try:
r = await client.post(
f"{BASE}/chat/completions",
headers={"Authorization": f"Bearer {KEY}"},
json={"model": MODEL, "max_tokens": MAX_TOK,
"messages": [{"role": "user", "content": PROMPT}]},
)
if r.status_code in (429, 500, 502, 503, 504) and attempts < 3:
await asyncio.sleep(0.5 * attempts)
continue
r.raise_for_status()
u = r.json().get("usage") or {}
return {"s": time.perf_counter() - t0, "attempts": attempts,
"tin": u.get("prompt_tokens"), "tout": u.get("completion_tokens")}
except Exception:
if attempts >= 3:
raise
await asyncio.sleep(0.5 * attempts)
async def level(n):
sem = asyncio.Semaphore(n)
async with httpx.AsyncClient(timeout=120) as client:
out = await asyncio.gather(*(one(client, sem) for _ in range(PER_LEVEL)),
return_exceptions=True)
ok = [o for o in out if isinstance(o, dict)]
if not ok:
return n, 0, None, None, None, None
lat = sorted(o["s"] for o in ok)
p50, p95 = statistics.median(lat), lat[max(0, int(len(lat) * 0.95) - 1)]
retries = sum(o["attempts"] - 1 for o in ok) / len(ok)
tin = sum(o["tin"] or 0 for o in ok)
tout = sum(o["tout"] or 0 for o in ok)
token_cost = ((tin / 1e6) * PRICE_IN + (tout / 1e6) * PRICE_OUT) * (1000 / len(ok))
wait_cost = (sum(lat) / len(lat)) * (HOURLY_RATE / 3600) * 1000
return n, len(ok) / len(out), p50, p95, retries, token_cost + wait_cost
async def main():
print("conc,ok_rate,p50_s,p95_s,retries_per_req,eff_cost_per_1k")
for n in LEVELS:
print("%.2f" % n if False else n, *[("—" if v is None else round(v, 4)) for v in (await level(n))[1:]], sep=",")
asyncio.run(main())
Run it twice. Once against the free path, once against the paid path with the same prompt and the same PER_LEVEL. The absolute latencies matter less than the slope. If p95 grows roughly linearly with concurrency, you are queueing behind other people and the queue is not yours to size. If p95 stays flat until some level and then jumps, you have found the ceiling of the shared pool and you now know it by number instead of by vibe.
The eff_cost_per_1k column is where the argument actually gets settled. It charges token spend at your paid rates and wait time at your loaded hourly rate, which is the honest trade: a request that finishes in 0.4 seconds and one that finishes in nine seconds are not the same product, even when both cost zero tokens. Set PRICE_IN and PRICE_OUT to zero for the free path and you will see the crossover anyway, because wait_cost does not go to zero when the price does.
Three readings of the table change what you ship. If the crossover sits above your real peak concurrency, free is the right default and you should stop worrying about it. If the crossover sits below your peak, free is fine for batch and background work and wrong for anything a human is waiting on. If retries per request climbs with concurrency, stop reading the table as a cost question — that is a correctness question wearing a cost question's clothes, and a retry storm on a shared pool can outlast your deadline.
This is the part the volume math hides. Retries are not free capacity doubled; they are the same request billed twice in time, and time is the resource that free tiers are actually short of. A pool that answers every request eventually is not the same as a pool that answers every request in time for your build to pass.
Where does MonkeyCode fit in this? The project is described by its operator as open source, and it offers two things that make a sweep like the one above cheap to run: free model access and a free server option. That combination matters for the harness specifically — you can put sweep.py on the free box and point it at the free endpoint, so the measurement costs you an afternoon rather than an invoice.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
But run the sweep before you bet a deadline on it, not after. The interesting number is not "what does a token cost today" — it is "at what concurrency does this stop being the right tool", and that number belongs to your workload, your prompt lengths, and your latency tolerance. If you want a place to start measuring, the project's own docs are the place to confirm current terms, because availability and model rosters move faster than any article about them.
Some honest limits on all of this. The harness measures one prompt shape at one size; long prompts and long completions will bend the curve differently. Token counts come back in usage on most compatible endpoints but not all, and when they are missing the script silently prices them at zero — check that column before you trust the total. Your HOURLY_RATE is a judgment call, and picking it too high will make every free option look bad. And a shared free pool is shared: your 3 a.m. sweep is not your 3 p.m. reality.
Do not use this approach for interactive latency budgets you have already promised to users, for anything where a mid-transaction failure costs real money, or for compliance workloads where you need a contract rather than a convenience. Free capacity is a good default for batch jobs, internal tooling, and the measurement harness itself. It is a bad bet the moment a human is staring at a spinner, or the moment a retry loop can eat the margin the free tier just gave you.
Measure the bend. Then choose the side of it you can defend.
Top comments (0)