Gate Jobs on Tokens per Accepted Artifact Before Retries Eat the Budget

Last Tuesday our batch lane burned 3.1 million tokens before 09:00.
Only forty-nine of sixty jobs produced an artifact we could ship.

Queue utilization looked healthy at 41%. The token graph went up 4x.
So which operational action follows from that evidence? Not a bigger queue. A budget gate.

Here is the workflow I now run before adding any free capacity to a lane.

The contradiction metric: tokens per accepted artifact

Utilization measures occupancy. It does not measure yield.
A retry loop can keep a lane busy while producing nothing.

Track one ratio instead:

• Numerator: every token you spend, including failed calls and rejected outputs.
• Denominator: artifacts that passed your checks and shipped.

Call it TPA (tokens per accepted artifact). Put it next to wall time.
When TPA rises while utilization stays flat, you are paying for retries, not throughput.

Two-phase reservation: price retries before you send them

Most teams cap tokens per request. That cap is useless under retries.
Three attempts each under the cap still triples your spend.

So reserve for the worst case, then settle on reality.
The gate below is plain Python with no dependencies. It is a runnable example, not a production library.

# budget_gate.py - two-phase token reservation for batch jobs
import json, time
from dataclasses import dataclass, field


class BudgetExceeded(Exception):
    pass


@dataclass
class Reservation:
    job_id: str
    lane: str
    reserved: int
    spent: int = 0
    attempts: int = 0
    opened_at: float = field(default_factory=time.time)


class TokenLedger:
    def __init__(self, cap_per_job: int, path: str = "ledger.jsonl"):
        self.cap = cap_per_job
        self.path = path
        self.open = {}

    def reserve(self, job_id: str, lane: str, estimate: int) -> Reservation:
        if estimate > self.cap:
            raise BudgetExceeded(f"{job_id}: estimate {estimate} > cap {self.cap}")
        res = Reservation(job_id=job_id, lane=lane, reserved=estimate)
        self.open[job_id] = res
        return res

    def charge(self, job_id: str, tokens: int) -> None:
        res = self.open[job_id]
        res.attempts += 1
        res.spent += tokens
        if res.spent > res.reserved:
            self.close(job_id, accepted=False, reason="over_reservation")
            raise BudgetExceeded(f"{job_id}: spent {res.spent} > reserved {res.reserved}")

    def close(self, job_id: str, accepted: bool, reason: str) -> None:
        res = self.open.pop(job_id, None)
        if res is None:
            return
        record = {
            "job_id": res.job_id, "lane": res.lane,
            "reserved": res.reserved, "spent": res.spent,
            "attempts": res.attempts, "accepted": accepted,
            "settle_reason": reason,
            "wall_ms": int((time.time() - res.opened_at) * 1000),
        }
        with open(self.path, "a") as fh:
            fh.write(json.dumps(record) + "\n")

The reservation covers retries. That is the whole point.

# runner.py - pre-pay for max_attempts, settle once
import time


def backoff(attempt: int) -> float:
    return min(2 ** attempt * 0.5, 8.0)


def run_job(ledger, job, client, max_attempts: int = 3):
    per_call = job.prompt_tokens + job.max_output_tokens
    ledger.reserve(job.id, lane=job.lane, estimate=per_call * max_attempts)
    last = None
    for attempt in range(1, max_attempts + 1):
        try:
            out = client.complete(job.prompt, max_tokens=job.max_output_tokens)
            # Assumption: the provider bills prompt tokens even on a failed call.
            # Verify your provider's billing; drop this charge if it does not.
            ledger.charge(job.id, out.usage_total)
            if passes_checks(out.text):
                ledger.close(job.id, accepted=True, reason="accepted")
                return out.text
            ledger.close(job.id, accepted=False, reason="rejected_by_checks")
            return None
        except Transient as exc:
            ledger.charge(job.id, job.prompt_tokens)
            last = exc
            time.sleep(backoff(attempt))
    ledger.close(job.id, accepted=False, reason=f"retries_exhausted:{last}")
    return None

Two properties matter more than the code.
Every job has a pre-paid ceiling, and every settled job leaves one JSONL record.

Read the ledger in one command

jq -s '
  {spent: (map(.spent) | add // 0),
   accepted: (map(select(.accepted)) | length),
   retried: (map(select(.attempts > 1)) | length)}
  | . + {tokens_per_accepted:
         (if .accepted == 0 then null else (.spent / .accepted) end)}' \
  ledger.jsonl

Expected output on a healthy lane, clearly labeled as expected, not measured by me on your workload:

{"spent": 1840000, "accepted": 49, "retried": 7, "tokens_per_accepted": 37551}

Now you have a number you can put in a review, an alert, or a capacity plan.

Run the shadow week on the cheapest box you have

You do not need production capacity to get this number.
I run the gate in shadow mode on a small free instance first, because shadow mode only needs to observe.

The MonkeyCode project advertises free model access, with the operator citing a 10M-token allowance, plus a free server option. I have not verified today's quotas, model list, or hardware.
Treat both as current-terms claims and read the project page before you size anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Shadow mode rules, and they are strict:

1. The gate computes reservations and logs them. It never blocks a job.
2. The real client still runs where it runs today. Nothing moves.
3. After seven days, compare tokens_per_accepted against your paid baseline.
4. Only then decide whether the free lane is worth enforcing.

Do not put a latency-critical path on a free box to "see how it goes".
Free capacity is a measurement instrument here, not a serving tier.

Fault injection before you trust the gate

A gate you never failed is a guess. Force failures locally.

# test_gate.py - run with: python -m pytest -q test_gate.py
import pytest
from budget_gate import BudgetExceeded, TokenLedger


class Transient(Exception):
    pass


class FlakyClient:
    def __init__(self, fail_first: int):
        self.fail_first = fail_first
        self.calls = 0

    def complete(self, prompt, max_tokens):
        self.calls += 1
        if self.calls <= self.fail_first:
            raise Transient("429 from upstream")
        return type("Out", (), {"text": "ok", "usage_total": 900})()


def test_retries_are_pre_paid(tmp_path):
    ledger = TokenLedger(cap_per_job=10000, path=str(tmp_path / "l.jsonl"))
    job = type("Job", (), {"id": "j1", "lane": "free",
                            "prompt_tokens": 800, "max_output_tokens": 400,
                            "prompt": "x"})()
    out = run_job(ledger, job, FlakyClient(fail_first=2), max_attempts=3)
    assert out == "ok"
    assert ledger.open == {}


def test_runaway_is_rejected(tmp_path):
    ledger = TokenLedger(cap_per_job=10000, path=str(tmp_path / "l.jsonl"))
    job = type("Job", (), {"id": "j2", "lane": "free",
                            "prompt_tokens": 800, "max_output_tokens": 400,
                            "prompt": "x"})()
    out = run_job(ledger, job, FlakyClient(fail_first=99), max_attempts=3)
    assert out is None
    assert ledger.open == {}

Assert three things after every run: no open reservations, spend never exceeds reservation, and the settle reason is populated.
An empty open dict is your leak detector.

Telemetry fields to emit

Six fields carry the whole diagnosis. Ship them on every settle.

• lane - which capacity class ran the job.
• reserved vs spent - reservation accuracy over time.
• attempts - retry distribution, not just the mean.
• accepted and settle_reason - why the tokens vanished.
• wall_ms - p95 against your deadline slack.
• tokens_per_accepted - the ratio you alert on.

Alert on the ratio, not on raw spend. Raw spend rises with useful work too.

When free capacity is the wrong bet

This is the part most write-ups skip. Free capacity has a cost, and it is usually wall time.
Use this table as a starting threshold set, then tune with your own ledger.

Signal over 7 days	Free lane is fine	Move work off the free lane
Retry rate	under 15% and flat	over 30%, or rising week over week
TPA vs paid baseline	within 1.2x	above 2x
p95 wall time	under 40% of deadline slack	approaching 70% of slack
Human review load	sampled review	every artifact needs review
Job side effects	idempotent, cheap to re-run	writes, payments, sends
Traffic shape	batch, tolerant of delay	interactive or user-facing

Two rows kill more free-lane plans than the rest. Non-idempotent writes and interactive latency.
No amount of saved tokens pays for a duplicate side effect.

Also price the human time. If a reviewer spends four minutes per artifact, TPA is not your biggest cost.

Limitations and who should not use this

• Hard deadline SLOs under five minutes. Reservation adds a write and a settle.
• Data that cannot leave your network. A hosted free tier is not the place to test it.
• Teams without per-call usage numbers. You cannot settle what you cannot measure.
• Jobs whose failures have side effects. Fix idempotency before you optimize spend.
• Workloads where output length is unbounded. Reserve a hard max_output_tokens first.

This gate also cannot tell you whether an artifact is good. It only prices what you already accepted.

Rollback and cleanup

Ship the gate with a mode switch, and know the switch before you need it.

# shadow: log only, never block
BUDGET_MODE=shadow python runner.py
# enforce: block over-cap jobs
BUDGET_MODE=enforce BUDGET_CAP_PER_JOB=12000 python runner.py

Rollback is one environment change: set BUDGET_MODE=shadow and restart.
No code revert, no ledger migration, no data loss.

Then clean up:

• Rotate ledger.jsonl daily. Sum the ratio, then archive.
• Add a reaper for reservations older than your job timeout. Dead workers leak reservations.
• Keep the fault-injection tests in CI. A retry change that skips the charge line is silent theft.

The one-line takeaway

Cap tokens per accepted artifact, not per request.
Then free capacity becomes a measured bet instead of a hopeful one.

If you want to reproduce the shadow week with no spend, the free model access and free server option are enough to collect the ledger above. Just verify the current terms yourself before you plan around them.