Free Token Budgets Hide Retry Cost: Keep a Per-Task LLM Ledger

It is 02:40 and your nightly enrichment job is 40 minutes behind. The page fires on queue age. No error rate moved.

You moved that job to free capacity last week. Spend went to zero. Throughput went with it. Now you must pick: revert, or wait it out.

This is the ordinary shape of a free-tier incident. The invoice is a terrible control signal. Tokens and seconds per successful task are the real ones.

Free capacity changes cost, not work

A retry is real work. A 503 still burned your wall clock. It may also have burned tokens before failing.

So two numbers move in opposite directions. Cost per token drops to zero. Cost per successful task can rise without bound.

You cannot see that gap without per-task accounting. Aggregate dashboards will not show it either. They average the failures away.

Declare the workload before you decide anything

Write these five things down first. Comparisons without them are noise.

1. Task shape: prompt tokens, completion tokens, output format, and strictness of parsing.
2. Concurrency: workers in flight and any per-key rate limits you already know about.
3. Retry policy: max attempts, backoff curve, jitter, and which errors are retryable at all.
4. Deadline: the SLO in minutes, plus the current queue age at decision time.
5. Failure taxonomy: 429, 5xx, connect timeout, read timeout, malformed output, refusal.

Step 5 is where most teams lose money. Retrying a malformed output or a refusal is pure burn. It can never succeed without changing the prompt.

A per-task ledger you can run today

The script below wraps your call site. It appends one JSON line per task. It is provider-agnostic and talks to an OpenAI-compatible endpoint.

#!/usr/bin/env python3
"""ledger.py - per-task attempt accounting for AI batch work."""
import json, os, time, urllib.error, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ.get("LLM_MODEL", "default")
LEDGER = os.environ.get("LEDGER_PATH", "ledger.jsonl")
RETRYABLE = {408, 429, 500, 502, 503, 504, 599}


def est_tokens(text: str) -> int:
    """Coarse estimate. Swap in your tokenizer before trusting totals."""
    return max(1, len(text) // 4)


def call(prompt: str, timeout: float = 45.0):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read())
    return payload["choices"][0]["message"]["content"], payload.get("usage", {})


def run_task(task_id, prompt, queue_wait_s=0.0, deadline_s=900, max_attempts=3):
    t0 = time.monotonic()
    rec = {
        "task_id": task_id, "ts": time.time(), "queue_wait_s": queue_wait_s,
        "attempts": 0, "prompt_tokens": 0, "completion_tokens": 0,
        "reported_tokens": 0, "attempt_latency_s": [], "status": "ok",
        "last_error": None, "deadline_s": deadline_s,
    }
    for n in range(1, max_attempts + 1):
        if time.monotonic() - t0 + queue_wait_s > deadline_s:
            rec["status"] = "deadline_exceeded"
            rec["last_error"] = "deadline_slack_exhausted"
            break
        rec["attempts"] = n
        rec["prompt_tokens"] += est_tokens(prompt)
        a0 = time.monotonic()
        try:
            text, usage = call(prompt)
            rec["attempt_latency_s"].append(round(time.monotonic() - a0, 3))
            rec["completion_tokens"] += usage.get("completion_tokens") or est_tokens(text)
            rec["reported_tokens"] += usage.get("total_tokens") or 0
            break
        except urllib.error.HTTPError as e:
            rec["attempt_latency_s"].append(round(time.monotonic() - a0, 3))
            rec["last_error"] = f"http_{e.code}"
            if e.code not in RETRYABLE:
                rec["status"] = "failed"
                break
        except (urllib.error.URLError, TimeoutError, OSError) as e:
            rec["attempt_latency_s"].append(round(time.monotonic() - a0, 3))
            rec["last_error"] = type(e).__name__
        time.sleep(min(2 ** n, 20) * (0.5 + 0.5 * (time.time() % 1)))
    else:
        rec["status"] = "failed"
    rec["elapsed_s"] = round(time.monotonic() - t0, 3)
    rec["total_tokens"] = rec["prompt_tokens"] + rec["completion_tokens"]
    with open(LEDGER, "a") as fh:
        fh.write(json.dumps(rec) + "\n")
    return rec

Run it against your own batch

export LLM_BASE_URL="https://your-endpoint.example/v1"
export LLM_API_KEY="..."
export LEDGER_PATH="ledger.jsonl"

python -c "from ledger import run_task; run_task('t-0001', 'summarize: <text>')"

Aggregate retry amplification

jq -r '[.attempts, .total_tokens, .queue_wait_s, .elapsed_s, .status] | @tsv' ledger.jsonl \
| awk '{a+=$1; tok+=$2; q+=$3; e+=$4; n++; if ($5=="ok") ok++}
       END {printf "tasks=%d ok=%d attempts/success=%.2f tokens/task=%.0f queue_p50~=%.1fs elapsed_p50~=%.1fs\n",
                    n, ok, a/ok, tok/n, q/n, e/n}'

Add a status breakdown, because failures still burn budget:

jq -r '.status' ledger.jsonl | sort | uniq -c | sort -rn
jq -r 'select(.attempts>1) | "\(.last_error)"' ledger.jsonl | sort | uniq -c | sort -rn

Example output (labeled, not a measurement)

The block below is illustrative formatting only. It is not a benchmark and your numbers will differ.

tasks=500 ok=482 attempts/success=1.31 tokens/task=2140 queue_p50~=38.0s elapsed_p50~=21.5s
    482 ok
     18 failed
   22 http_429
   11 http_503

Read 1.31 as the tax. Three out of every ten successes bought an extra attempt.

Three thresholds that flip the decision

1. Attempts per success

Keep this under your own tolerance. Many teams pick 1.2 because retry overhead plus queue growth eats the savings fast. Above it, free capacity is not cheaper. It is just slower.

2. Queue age against deadline slack

Define slack as deadline_s - elapsed_p95. Then compare queue_p95 to it.

If queue age exceeds roughly a quarter of your slack, you have no room for one retry storm. Move the critical path back to capacity you control.

3. Token burn against the allowance

Treat an allowance as a burn-rate budget, not a target. Samples every five minutes:

jq -r 'select(.ts > (now - 300)) | .total_tokens' ledger.jsonl | awk '{s+=$1} END {print s/300, "tokens/sec"}'

If that rate reaches a meaningful fraction of the allowance while attempts/success stays elevated, cap concurrency. Do not add workers.

Decision table

Observed signal	Likely cause	Action
attempts/success 1.0, queue age rising	concurrency too low	raise workers, watch 429s
attempts/success > 1.3, 429 dominant	admission control missing	add token-bucket limiter, lower workers
attempts/success > 1.3, 5xx dominant	provider-side instability	keep batch on paid path, retry later
failures with malformed_output	prompt or parser, not capacity	fix prompt, do not retry
queue_p95 > 25% of slack	deadline risk	revert critical path, keep tail on free
token rate near allowance, low attempts	genuinely cheap workload	keep it there, record the reason

That last row matters. Some workloads belong on free capacity. You need evidence, not a hunch.

Failure handling and rollback

Ship a kill switch and a drain path before you migrate anything.

1. Cap max_attempts at 3 in code, configurable by env, never by caller.
2. Route exhausted tasks to a dead-letter file with the full ledger record.
3. Add FREE_TIER_ENABLED=0 to bypass the new path without a deploy.
4. Shadow first: run both paths for one window and compare attempts/success.
5. Drain: stop dequeuing, let in-flight tasks finish, then flip the flag.
6. Keep the old endpoint config for one week after cutover.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator states that MonkeyCode provides free model access and a free server option, with a 10M token allowance. Treat those as availability claims from the vendor, not as measurements. Substituting them into this ledger produces comparable numbers, which is the point.

A free server option removes one objection: you can run the ledger collector and a shadow batch without a new invoice. Run the shadow window there first. If attempts/success stays near 1.0 and queue age fits your slack, promote the workload. If it does not, you learned that for the cost of a JSON file.

Keep the ledger provider-neutral. The file format above does not name a vendor for a reason. When a cheaper path appears next quarter, you only change the endpoint.

Limitations and who should not use this

• Token counts here are estimates. Use a real tokenizer before you forecast spend.
• Single-process file appends will not survive high concurrency. Ship records to a log collector or database instead.
• A per-task ledger is not distributed tracing. Pair it with span IDs if you need causality.
• If your SLO has hard, contractually visible deadlines, free capacity is the wrong bet regardless of price. Keep the critical path where you control latency.
• If you cannot instrument the call site, do not migrate. Measure first.
• If your payloads are regulated, review the data path before sending anything to a new endpoint.
• If your workload is a single interactive request with a human waiting, retry budgets rarely pay off. Latency is the product.

What to do next

Pick one batch job. Wrap it with the ledger for one week. Then compare attempts/success, queue age against slack, and token burn rate.

Those three numbers will tell you whether free capacity is a saving or a deferral. Which one of them is currently invisible in your dashboards?