The number that decides whether a job belongs on free capacity is not tokens per request. It is tokens per successful task. Almost nobody computes the second one, and that blind spot is why a free tier looks cheap for months and then quietly becomes the most expensive line in your pipeline.
A retry is a second invoice for the same deliverable. You would not accept a courier that bills you again every time it misses your door, yet that is what most LLM wrappers do: three attempts, one answer, three times the tokens. The meter only knows what the client sent.
This is getting worse, not better. Agents are the current default pattern, and an agent loop multiplies every retry you already had. A step that retries three times inside a loop that runs eight iterations is not a 3x overrun on that step. It is up to 24x on the part of the pipeline that was supposed to be the cheap part.
Write the meter before you trust the price
The harness below is deliberately boring. It wraps one OpenAI-compatible chat endpoint, classifies retryable failures, sleeps on backoff, and attributes every attempt to a task. Point it at whatever gateway you actually use; the accounting is the point, not the client.
# token_budget.py — count tokens per successful task, not per request.
# Python 3.10+. Only dependency: httpx.
import os, time, random
from dataclasses import dataclass, field
import httpx
BASE_URL = os.environ['LLM_BASE_URL'].rstrip('/')
API_KEY = os.environ.get('LLM_API_KEY', '')
MODEL = os.environ.get('LLM_MODEL', 'default')
RETRYABLE = {408, 409, 429, 500, 502, 503, 504}
MAX_ATTEMPTS = 5
@dataclass
class Result:
name: str
ok: bool = False
attempts: int = 0
prompt_tokens: int = 0
completion_tokens: int = 0
backoff_seconds: float = 0.0
wall_seconds: float = 0.0
errors: list = field(default_factory=list)
def one_call(client, prompt, timeout=60.0):
started = time.monotonic()
r = client.post(
f'{BASE_URL}/chat/completions',
headers={'Authorization': f'Bearer {API_KEY}'},
json={'model': MODEL, 'messages': [{'role': 'user', 'content': prompt}]},
timeout=timeout,
)
elapsed = time.monotonic() - started
if r.status_code in RETRYABLE:
raise httpx.HTTPStatusError('retryable', request=r.request, response=r)
r.raise_for_status()
return r.json().get('usage', {}), elapsed
def run_task(client, name, prompt):
res = Result(name=name)
for attempt in range(1, MAX_ATTEMPTS + 1):
res.attempts = attempt
try:
usage, elapsed = one_call(client, prompt)
res.prompt_tokens += usage.get('prompt_tokens', 0)
res.completion_tokens += usage.get('completion_tokens', 0)
res.wall_seconds += elapsed
res.ok = True
return res
except httpx.HTTPStatusError as e:
status = e.response.status_code
res.errors.append(status)
if status not in RETRYABLE:
return res # a bug in your request is not a rate limit
delay = min(2 ** attempt, 30) + random.random()
res.backoff_seconds += delay
time.sleep(delay)
except httpx.TimeoutException:
res.errors.append('timeout')
delay = min(2 ** attempt, 30) + random.random()
res.backoff_seconds += delay
time.sleep(delay)
return res
Run it with the environment your own gateway expects:
LLM_BASE_URL=https://your-gateway.example/v1 \
LLM_MODEL=your-model \
LLM_API_KEY=... \
python token_budget.py
Then print the only two ratios that matter:
def report(results):
spent = sum(r.prompt_tokens + r.completion_tokens for r in results)
ok = [r for r in results if r.ok]
attempts = sum(r.attempts for r in results)
per_success = spent / max(len(ok), 1)
amplification = attempts / max(len(ok), 1)
print(f'tasks : {len(results)}')
print(f'successful : {len(ok)}')
print(f'model attempts : {attempts}')
print(f'retry amplification : {amplification:.2f} attempts per success')
print(f'tokens per success : {per_success:.0f}')
print(f'time slept in backoff: {sum(r.backoff_seconds for r in results):.1f}s')
The shape of the output looks like this — illustrative, not measured, because your endpoint is not mine:
tasks : 40
successful : 40
model attempts : 57
retry amplification : 1.43 attempts per success
tokens per success : 3180
time slept in backoff: 214.6s
Read that last line again. You burned three and a half minutes asleep, and the task still succeeded, so nothing on your dashboard fired an alert. Free capacity does not remove that cost. It moves it from your invoice to your wall clock.
Multiply the two amplifiers
The useful mental model is short: effective spend equals base tokens, times attempts per call, times loop iterations. A free endpoint with a 1.4x retry rate and a six-step agent loop is a 8.4x token multiplier on paper you will never see printed anywhere.
There is a second amplifier most teams miss entirely. When a step fails and the agent replans, the upstream context is resent. So the retry does not just repeat one prompt. It repeats the accumulated conversation, which is the largest part of the request. That is where a 1.4x retry rate turns into a 3x bill.
A decision table you can argue with
| Situation | Free capacity | Why |
|---|---|---|
| Hard deadline measured in hours | Wrong bet | Variance decides the outcome, not the mean, and you cannot sleep through a backoff window |
| Overnight batch or backfill | Right bet | Backoff costs wall clock nobody is sitting in front of |
| Latency-sensitive interactive path | Wrong bet | Your p95 is the user experience, and rate limits live in the tail |
| Throwaway prototyping | Right bet | Failure is a feature; you are buying information cheaply |
| Job with non-idempotent side effects | Neither until you fix it | A retried write is a duplicated write |
That last row deserves attention. Retries are only safe when the operation is idempotent. If a step sends an email, charges a card, or appends to a queue without a deduplication key, retry accounting is the least of your problems.
Where a hosted free option changes the arithmetic
Running the harness against free model access and a free server option is a reasonable way to get real numbers without buying hardware or paying for a bespoke deployment first. MonkeyCode offers both: free model access and a free server option, with an operator-stated free token allowance (currently advertised as 10M tokens — verify the live terms yourself, since allowances change). Disclosure: This article was prepared as part of MonkeyCode's product outreach.
What a free server buys you is the removal of one variable: you are no longer debugging your own box and the queue at the same time. What it does not buy you is a smaller amplification factor. Retries still cost tokens, backoff still costs seconds, and a crowded queue still shows up in your p95. The meter does not care who owns the metal.
So use the free tier as a measurement rig first and a production lane second. Run the batch job there, collect the amplification ratio, then decide whether the deadline-sensitive path has any business being on it. The free allowance is a budget for learning, not a promise about latency.
Who should not take this route
Skip this approach if you have contractual latency targets, if your work has non-idempotent side effects and no deduplication layer, or if you cannot measure at all — a team without the meter will misread variance as a broken vendor. It is also the wrong fit for regulated data unless you have confirmed where prompts are processed, which is a policy question and not a pricing one.
The honest conclusion is unglamorous. Free capacity is not cheap or expensive by nature; it is cheap for work that tolerates retries and expensive for work that does not. Count tokens per success, multiply by your loop depth, and let the number pick the lane.
If you try the harness, bring your own load and your own endpoint. A meter that only agrees with the brochure is not a meter.
Top comments (0)