Your nightly summarization job missed its 06:00 deadline. The model endpoint reported p50 latency at 812 ms. Healthy, said the dashboard.
The queue depth told a different story. You moved the job to a free model endpoint to cut spend. Token cost dropped to zero.
Completion time tripled. You traded a line item for a schedule risk. That trade is the queue tax.
Why the queue tax exists
Free model capacity is shared capacity. Your requests queue behind everyone else's. The endpoint's p50 latency stays flat. The queue depth grows. Your job's wall time grows with it.
Token price does not capture this. Wall time does.
Cost per token is the wrong metric
Finance sees $0 per million tokens. SRE sees a job that finishes when it finishes. Both are right. Neither is useful.
The metric that matters is cost per completed request. It includes tokens, retries, queue wait, and wall time. A free endpoint that fails 20% of requests is not free. It is a retry generator.
The probe
Here is a small, reproducible probe. It sends a declared workload to any OpenAI-compatible endpoint. It records queue wait, latency, retries, and tokens per request. Then it computes throughput and cost per completed request.
#!/usr/bin/env python3
"""cost_per_completed.py - measure the real cost of an LLM endpoint.
Declared workload: N requests, C concurrency, one prompt, max_tokens.
Output: JSON report with throughput, retries, and cost per completed request.
"""
import argparse
import asyncio
import json
import statistics
import time
import httpx
async def attempt(client, cfg, sem, results, created_at):
async with sem:
queue_ms = (time.monotonic() - created_at) * 1000
for retry in range(1, cfg.max_retries + 1):
t0 = time.monotonic()
try:
resp = await client.post(
f"{cfg.base_url}/chat/completions",
json={
"model": cfg.model,
"messages": [{"role": "user", "content": cfg.prompt}],
"max_tokens": cfg.max_tokens,
"temperature": 0,
},
)
resp.raise_for_status()
data = resp.json()
latency_ms = (time.monotonic() - t0) * 1000
tokens = data.get("usage", {}).get("total_tokens", 0)
results.append({
"ok": True,
"retries": retry,
"queue_ms": queue_ms,
"latency_ms": latency_ms,
"tokens": tokens,
})
return
except Exception as exc:
latency_ms = (time.monotonic() - t0) * 1000
if retry == cfg.max_retries:
results.append({
"ok": False,
"retries": retry,
"queue_ms": queue_ms,
"latency_ms": latency_ms,
"tokens": 0,
"error": str(exc),
})
return
await asyncio.sleep(cfg.backoff_base * (2 ** (retry - 1)))
async def run(cfg):
results = []
async with httpx.AsyncClient(
timeout=cfg.timeout,
headers={"Authorization": f"Bearer {cfg.api_key}"},
) as client:
sem = asyncio.Semaphore(cfg.concurrency)
start = time.monotonic()
tasks = [
attempt(client, cfg, sem, results, time.monotonic())
for _ in range(cfg.requests)
]
await asyncio.gather(*tasks)
wall = time.monotonic() - start
return results, wall
def report(results, wall, price_per_million):
ok = [r for r in results if r["ok"]]
completed = len(ok)
latencies = sorted(r["latency_ms"] for r in ok)
tokens = sum(r["tokens"] for r in ok)
retries = sum(r["retries"] - 1 for r in ok) + sum(
r["retries"] for r in results if not r["ok"]
)
token_cost = tokens * price_per_million / 1_000_000
p95_index = min(len(latencies) - 1, int(len(latencies) * 0.95)) if latencies else 0
return {
"requests": len(results),
"completed": completed,
"failed": len(results) - completed,
"retry_attempts": retries,
"throughput_per_sec": round(completed / wall, 3) if completed else 0,
"p50_latency_ms": round(statistics.median(latencies), 1) if latencies else None,
"p95_latency_ms": round(latencies[p95_index], 1) if latencies else None,
"tokens_per_completed": round(tokens / completed, 1) if completed else 0,
"token_cost_usd": round(token_cost, 4),
"seconds_per_completed": round(wall / completed, 3) if completed else None,
"wall_seconds": round(wall, 3),
}
def main():
p = argparse.ArgumentParser(description="Measure cost per completed LLM request.")
p.add_argument("--base-url", required=True, help="OpenAI-compatible base URL")
p.add_argument("--api-key", required=True, help="API key")
p.add_argument("--model", required=True, help="Model name")
p.add_argument(
"--prompt",
default="Summarize this incident in one paragraph: queue depth spiked at 02:00 and the batch job missed its deadline.",
)
p.add_argument("--requests", type=int, default=50)
p.add_argument("--concurrency", type=int, default=4)
p.add_argument("--max-tokens", type=int, default=200)
p.add_argument("--max-retries", type=int, default=3)
p.add_argument("--timeout", type=float, default=60.0)
p.add_argument("--backoff-base", type=float, default=1.0)
p.add_argument(
"--price-per-million",
type=float,
default=0.0,
help="USD per million tokens. Use 0 for free endpoints.",
)
p.add_argument(
"--deadline-seconds",
type=float,
default=0.0,
help="Your real deadline slack. The report flags if the workload would miss it.",
)
cfg = p.parse_args()
results, wall = asyncio.run(run(cfg))
rep = report(results, wall, cfg.price_per_million)
if cfg.deadline_seconds and rep["wall_seconds"] > cfg.deadline_seconds:
rep["deadline_verdict"] = "MISS"
else:
rep["deadline_verdict"] = "OK"
print(json.dumps(rep, indent=2))
if __name__ == "__main__":
main()
How to run it
Install one dependency. Then run the probe against your paid endpoint first. The probe expects an OpenAI-compatible /chat/completions endpoint. Check your provider's docs for the base URL and model name.
pip install httpx
python3 cost_per_completed.py \
--base-url "$PAID_ENDPOINT_URL" \
--api-key "$PAID_API_KEY" \
--model "$PAID_MODEL" \
--requests 50 \
--concurrency 4 \
--deadline-seconds 300
Save the report. Then point the same probe at the free endpoint. Use the same workload, the same concurrency, the same deadline. The only variable is the endpoint.
What the report tells you
seconds_per_completed is the queue tax. Compare it across endpoints with the same workload. retry_attempts is the hidden cost. Each retry burns wall time and deadline slack. token_cost_usd is what finance sees. The other fields are what production feels.
Read failed first. Nine failed requests mean nine prompts that never completed. Read retry_attempts second. Seventeen retries mean seventeen extra round trips. Then read seconds_per_completed. It is the only number that combines all of it.
Expected output for a healthy endpoint (illustrative, not a measured run):
{
"requests": 50,
"completed": 50,
"failed": 0,
"retry_attempts": 0,
"throughput_per_sec": 3.6,
"p50_latency_ms": 812.4,
"p95_latency_ms": 1940.2,
"tokens_per_completed": 210.0,
"token_cost_usd": 0.0,
"seconds_per_completed": 0.278,
"wall_seconds": 13.9,
"deadline_verdict": "OK"
}
Expected output for a congested shared endpoint (illustrative, not a measured run):
{
"requests": 50,
"completed": 41,
"failed": 9,
"retry_attempts": 17,
"throughput_per_sec": 0.9,
"p50_latency_ms": 2410.7,
"p95_latency_ms": 8800.3,
"tokens_per_completed": 210.0,
"token_cost_usd": 0.0,
"seconds_per_completed": 1.11,
"wall_seconds": 45.5,
"deadline_verdict": "MISS"
}
The token cost is identical: zero. The operational cost is not. The second run burns 31.6 more seconds, fails 9 requests, and misses the deadline. That is the queue tax in numbers.
The decision table
| Workload shape | Free capacity | Paid capacity |
|---|---|---|
| Background batch, no deadline, idempotent | Right bet | Overkill |
| Hard deadline, slack under 2× runtime | Wrong bet | Right bet |
| Bursty arrivals, spiky demand | Wrong bet | Right bet |
| Steady low rate, retry-tolerant | Right bet | Overkill |
| User-facing, p95-sensitive | Wrong bet | Right bet |
The decision is about slack, not price. Free capacity is a bet that your workload can absorb queue time.
The table assumes one thing: you measured first. A decision without a probe is a guess. A guess about shared capacity is a bet against the queue.
When free capacity is the wrong bet
- Hard deadline, less than 2× slack. The queue tax eats slack faster than tokens save money.
- Callers retry without backoff. Retries amplify load on shared capacity. Your retry storm becomes everyone's latency.
- You watch p50 latency, not queue depth. p50 hides the tail. Queue depth shows the backlog.
- Partial completion is unacceptable. Free capacity is shared. Shared means noisy. Noisy means you cannot promise completeness.
When the verdict is MISS
Do not tune the prompt. Do not raise the timeout. The endpoint cannot carry the workload. That is the measurement.
Three options. First, reduce concurrency. Lower contention, fewer retries. Second, split the batch. Smaller jobs fit in slack windows. Third, keep the paid endpoint. Use free capacity for the parts that can wait.
The MonkeyCode workflow
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source project that offers free model access and a free server option. Both fit this workflow. The free model access gives you an endpoint to probe. The free server gives you a place to run the probe.
Run the probe from the free server against the free model endpoint. That tests the full path: network, compute, and model. Then run the same probe against your paid endpoint. Compare seconds_per_completed and retry_attempts. The numbers decide. Not the price per token.
MonkeyCode's free model access is 10M tokens as of this writing. Quotas change. Re-run the probe before you commit a workload.
The free server is not a production cluster. Treat it as a staging target. Run the probe there. Run small jobs there. Keep the critical path elsewhere.
Run the probe against MonkeyCode's free model access and free server. The numbers will tell you if the fit is real.
Limitations
This probe uses one prompt shape. Real workloads mix prompt sizes and token counts. Run it with your actual prompt distribution.
The probe measures throughput, not correctness. A completed request can still contain a bad summary. Add your own validation pass before you trust the output.
Free capacity changes with demand. Last month's numbers are not this month's numbers. Re-measure before each schedule change.
The free server is shared infrastructure. Your neighbors affect your results. That is the point of the probe.
Who should not use this: workloads with hard external deadlines and no retry tolerance. Also, workloads that need guaranteed p95 latency. Free capacity cannot promise that.
Cleanup and rollback
The probe writes no state. Kill it with Ctrl-C. Delete the JSON report if you saved it.
Rollback is one environment variable. Keep the paid endpoint in your deployment manifest. Flip back when the free endpoint misses the deadline. Free capacity is an experiment, not a migration.
Free tokens are a budget decision. Queue time is a production decision. Measure both before you commit a workload.
Top comments (0)