Retry Amplification Is What Actually Burns Your Free Token Budget

At 09:12 the meter read 41k tokens per hour. By 11:30 it read 310k. Job count was flat. Traffic was flat. One prompt class was retrying itself into a hole.

Nothing in the dashboard named that. The graph showed tokens, and tokens looked like demand. They were not demand. They were a retry loop wearing demand's clothes.

Here is the guard I now put in front of any free-tier batch worker, and the thresholds that decide when free capacity stops being the right bet.

The number you were watching was wrong

Tokens per hour is an input metric. It tells you what you spent, not what you got.

Two lanes can burn identically and deliver wildly different value. Lane A: 900 jobs, 1 attempt each. Lane B: 300 jobs, 3 attempts each. Same token rate. Different product.

So track three numbers instead:

1. Retry amplification — attempts divided by distinct jobs.
2. Tokens per accepted artifact — total tokens divided by jobs that reached accepted.
3. Hours to empty — remaining allowance divided by the current burn rate.

Then compare hours-to-empty against deadline slack, not against a percentage bar. Utilization at 80% is fine with 40 hours of slack. It is fatal with 2.

Declare the workload before you trust any of it

This guard was built for one declared shape. Change the shape and you must re-measure.

• One batch worker process, 4 concurrent jobs.
• 200 mixed jobs per run, three prompt classes (summarize, classify, extract).
• Retry policy: 3 attempts, exponential backoff, no class separation.
• Meter reads an append-only attempt log, one JSON line per model call.

The log contract matters more than the code. If you do not log per-attempt token counts, you cannot separate demand from amplification later.

{"ts": 1761512400, "job_id": "j-8812", "class": "summarize",
 "attempt": 1, "prompt_tokens": 1800, "completion_tokens": 240,
 "outcome": "accepted"}

outcome is one of accepted, rejected, error. Write the line after the call returns, not before. A crash mid-call should leave no spend record you cannot verify.

The artifact: a burn-rate guard in one file

Drop this next to the worker. It reads the log, prints four numbers, and returns a verdict your admission hook can act on.

#!/usr/bin/env python3
"""burn.py - token burn-rate guard for a single AI batch worker."""
import argparse, json, sys
from collections import defaultdict


def load(path):
    rows = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def tokens(r):
    return r["prompt_tokens"] + r["completion_tokens"]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--log", required=True)
    ap.add_argument("--budget", type=int, required=True)
    ap.add_argument("--window", type=int, default=3600)
    ap.add_argument("--slack-hours", type=float, required=True)
    ap.add_argument("--amplification-cap", type=float, default=1.6)
    ap.add_argument("--per-artifact-cap", type=int, default=4000)
    args = ap.parse_args()

    rows = load(args.log)
    if not rows:
        sys.exit("empty log: nothing to meter")

    now = max(r["ts"] for r in rows)
    spent = sum(tokens(r) for r in rows)
    recent = [r for r in rows if r["ts"] >= now - args.window]
    rate = sum(tokens(r) for r in recent) / args.window  # tokens per second

    by_job = defaultdict(list)
    for r in rows:
        by_job[r["job_id"]].append(r)
    accepted = [j for j, rs in by_job.items()
                if any(r["outcome"] == "accepted" for r in rs)]

    amp = len(rows) / max(len(by_job), 1)
    per_artifact = spent / max(len(accepted), 1)
    remaining = args.budget - spent
    eta_h = (remaining / rate / 3600) if rate > 0 else float("inf")

    print(f"jobs={len(by_job)} attempts={len(rows)} accepted={len(accepted)}")
    print(f"spent={spent} remaining={remaining} rate={rate:.0f}tok/s")
    print(f"amplification={amp:.2f} tokens_per_accepted={per_artifact:.0f}")
    print(f"eta_hours={eta_h:.1f} slack_hours={args.slack_hours}")

    verdict = "ADMIT"
    if amp > args.amplification_cap:
        verdict = "CAP_ATTEMPTS"
    if per_artifact > args.per_artifact_cap:
        verdict = "QUARANTINE_CLASS"
    if eta_h < args.slack_hours:
        verdict = "HOLD_NEW_WORK"
    print(f"verdict={verdict}")


if __name__ == "__main__":
    main()

Run it from cron, or from the admission hook, every 60 seconds:

python3 burn.py \
  --log /var/log/jobs/attempts.jsonl \
  --budget 10000000 \
  --slack-hours 14 \
  --amplification-cap 1.6

Expected output for a synthetic log

I have not run this against a production lane. The block below is expected output for the synthetic 200-job log shipped with the script. Label it as such in your own runbook too.

jobs=200 attempts=331 accepted=173
spent=604200 remaining=9395800 rate=21tok/s
amplification=1.66 tokens_per_accepted=3492
eta_hours=124.3 slack_hours=14.0
verdict=CAP_ATTEMPTS

Read it in this order. amplification=1.66 says 158 of 331 attempts bought nothing. tokens_per_accepted=3492 says each shipped artifact carries almost one and a half failed attempts. The ETA is comfortable, and that is exactly the trap: you have tokens but not throughput.

Where the free server actually fits

The meter must run where the worker runs. A second box for observability turns a free lane into a paid one.

That is why I keep both the worker and burn.py on the same free server, sharing the same log file. The meter is stateless, so a restart costs you nothing.

MonkeyCode's free model access and free server option fit that shape. The free arrangement is an allowance of about 10M tokens, per the project's own materials — treat that figure as operator-supplied and re-check current limits before you size anything against it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A 10M-token allowance is a burst allowance, not a steady-state budget. At 21 tokens/second, it covers roughly five and a half days of continuous running. At 200 tokens/second, barely 14 hours. Never plan against the first number unless you have measured the second.

Which constraint is binding

Symptom	Binding constraint	Operational move
Amplification > 1.6, tokens per accepted flat	Retry policy	Cap attempts at 2, quarantine on exhaustion
Tokens per accepted up 20%, job count flat	Prompt or input size	Revert the prompt change before scaling
ETA < deadline slack, amplification < 1.2	Real demand	Reduce concurrency or move a class off the free lane
Queue age up, utilization low	Admission, not capacity	Check for a hold or a stalled worker

Utilization is the wrong threshold for the last row. Queue age against deadline slack tells you whether anyone cares about the wait.

When free capacity is the wrong bet

Walk away from the free lane in these cases:

• Your job class needs a hard p99 latency target. Shared capacity cannot promise it.
• Amplification stays above 1.6 after the attempt cap. You are paying retries with time you do not have.
• Tokens per accepted artifact exceeds your cost of a second attempt elsewhere.
• The work must survive a node restart mid-batch with no requeue.

The free lane is a cost probe. It tells you the shape of your workload cheaply. It is not a capacity commitment.

Failure handling, cleanup, rollback

Policy file read at admission:

retry:
  max_attempts: 2
  backoff_s: [4, 16]
  on_exhausted: quarantine      # dead partition, never auto re-enqueue
admission:
  hold_when: "verdict in {HOLD_NEW_WORK, QUARANTINE_CLASS}"
  recheck_s: 60

Rollback path, in order:

1. Set max_attempts back to its previous value and restart the worker.
2. Export the quarantine partition to cold storage.
3. Re-enqueue quarantined jobs with attempt reset only after the prompt fix ships.
4. Confirm amplification drops below 1.6 over two windows before re-enabling admission.

Cleanup is boring on purpose:

logrotate /etc/logrotate.d/jobs-attempts   # daily, keep 7, compress

Keep the attempt log longer than the job log. Cost questions always arrive after the jobs are gone.

Limitations and who should skip this

• EWMA-free extrapolation breaks on step changes. A prompt deploy invalidates the ETA instantly.
• Per-attempt token counts are only as good as your client. Streaming calls can under-report.
• Allowance terms change. Re-verify before you set a --budget value in CI.
• Interactive, latency-critical traffic should not use this guard. It optimizes spend, not response time.

Skip it entirely if you have tenant-level billing with per-request cost attribution. You already have better numbers than amplification.

If you want to try the workflow without provisioning anything, the free model access and free server option are enough to run one worker and this meter on the same box. Start with a 200-job synthetic batch and read the verdict before you point it at real traffic.