DEV Community: Quinn Li

Retries Are Rent

Quinn Li — Fri, 16 Oct 2026 05:21:41 +0000

Your retry loop does not bill you in tokens. It bills you in seconds you cannot buy back, and free capacity is the one lane where the second invoice is the only one that arrives. That is the whole argument. Everything below is a way to put a number on it before you hand a week of batch work to a queue you do not control.

Tokens are the visible line on the receipt. Wall clock is the line the vendor never prints. When you move a workload onto a free lane, the first line shrinks and the second one grows, and most teams never notice the trade because only one of the two numbers shows up in a dashboard.

The ledger you are not keeping

Every agent framework I read now ships a retry policy, and almost none of them log what a retry costs in time. So write the ledger yourself. The harness below is a proposal, not something I ran against a hosted endpoint, so treat the output as a placeholder until you point it at your own client.

# retry_ledger.py — proposal; run it against your own client before trusting it
from dataclasses import dataclass, field
from time import monotonic

RETRYABLE = {"rate_limited", "timeout", "upstream_5xx"}

@dataclass
class Attempt:
    queue_wait_s: float
    work_s: float
    tokens_in: int
    tokens_out: int
    outcome: str

@dataclass
class Task:
    name: str
    attempts: list = field(default_factory=list)

    @property
    def succeeded(self):
        return bool(self.attempts) and self.attempts[-1].outcome == "ok"

    @property
    def wall_s(self):
        return sum(a.queue_wait_s + a.work_s for a in self.attempts)

    @property
    def queue_share(self):
        return sum(a.queue_wait_s for a in self.attempts) / max(self.wall_s, 1e-9)

    @property
    def tokens(self):
        return sum(a.tokens_in + a.tokens_out for a in self.attempts)


def run_task(name, call, max_attempts=4, backoff=1.5):
    task = Task(name)
    delay = 1.0
    for _ in range(max_attempts):
        t0 = monotonic()
        result = call(name)          # your client returns outcome + timing + usage
        t1 = monotonic()
        task.attempts.append(Attempt(
            queue_wait_s=getattr(result, "queue_wait_s", 0.0),
            work_s=t1 - t0 - getattr(result, "queue_wait_s", 0.0),
            tokens_in=getattr(result, "tokens_in", 0),
            tokens_out=getattr(result, "tokens_out", 0),
            outcome=result.outcome,  # "ok" | "rate_limited" | "timeout" | "malformed" | "bad_input"
        ))
        if result.outcome == "ok":
            return task
        if result.outcome not in RETRYABLE:
            return task              # do not buy rent on a bug you can fix
        delay *= backoff
    return task

Two lines in that dataclass matter more than the rest. queue_share tells you how much of your latency was someone else's scheduling, and tokens counts every attempt, not just the successful one. Log both and the free-lane decision stops being a vibe.

Failure classes decide who pays

Retrying a bad_input error is the most expensive habit in this whole space. You pay queue wait, you pay tokens, and the answer cannot change. Treat malformed the same way until you have a repair step that actually alters the request.

Retries are rent because the landlord is the queue, not your code. A three-attempt success at 12 seconds of queue wait each costs you 36 seconds of clock whether or not a single token was billed. On an interactive path that is a product failure. On an overnight batch it is a rounding error.

Where the free lane actually wins

Free capacity is a scheduling choice, not a discount. It wins when your deadline is loose, your tasks are idempotent, and your failures are retryable. It loses when a human is watching the spinner, or when retry amplification pushes you past a wall-clock deadline that has money attached.

Workload shape	Free lane	Committed lane
Interactive edit, p99 under 2s	no	yes
Overnight batch, morning deadline	yes, if queue_share stays under ~50%	yes if it does not
Fan-out where 30%+ attempts are rate_limited	yes, with a jittered cap	yes
Anything with `bad_input` in the loop	fix the input first	fix the input first
Long cross-repo refactor you must restart often	risky: restarts multiply queue wait	yes

That table is a starting point, not a benchmark. Fill in your own columns from the ledger after a few hundred tasks.

Where MonkeyCode fits in the ledger

The project currently advertises free model access and a free server option, which makes it a reasonable candidate for the slow lane in the harness above: batch jobs where you care about the total and not the tail. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not going to quote a quota number here. Allowances, model availability, and server terms are set by the project and they move; read the current terms instead of a blog post, including this one. What matters for the ledger is only that you can point call() at it, capture queue wait if the client exposes timing, and count attempt-level tokens.

One natural way to start: take a batch you already run, cap max_attempts at two, run it twice — once on the lane you pay for, once on the free lane — and compare tokens / successes and wall_s / successes side by side. Two numbers, one afternoon.

Test plan you can run this week

Write a stub call() that returns rate_limited 30% of the time and ok otherwise, with a fixed 8-second simulated queue wait. Run 200 tasks and assert the ledger reports an amplification near 1.43 and a queue_share above 0.6. Then flip the stub to 2 seconds of queue wait and rerun. The amplification barely moves; the wall clock collapses.

That experiment is the point. Free capacity changes the constant, not the shape. If your retry temperature is wrong, a faster queue just gets you to the wrong answer sooner.

Limitations and who should skip this

Do not use the free lane for anything with a human on the other end, for regulated data whose residency rules you have not confirmed, or for workloads you must restart often. Do not use it when the deadline penalty exceeds the token savings by an order of magnitude — usually the case for paid client work.

And do not use any of this if you cannot state your per-success budget. A ledger without a target is a diary. Pick the number first, then let the queue wait tell you which lane you can afford.

Price the Run, Not the Token

Quinn Li — Thu, 15 Oct 2026 11:31:27 +0000

Price the Run, Not the Token

Your spend graph has two meters and the free one only covers half of them. Tokens are the meter you can see on a dashboard. Seconds are the meter your deadline actually reads. Cap the first and ignore the second, and you will ship late while the chart looks perfectly healthy.

This is the failure mode that shows up whenever free capacity is in the mix. Free tokens make attempts cheap, so people stop counting attempts. Attempts are not free in time.

Every attempt re-buys the context

A retry is not a replay from the last good word. In most agent loops you append the failure or the validator output and send the whole thing again. Attempt one costs your context. Attempt two costs your context plus the first error. Attempt three costs context plus two errors, and so on.

The token cost of a run therefore grows faster than the number of attempts, and the wall-clock cost grows with queue wait you do not control. Two retries on a shared free tier might be ninety seconds of generation and four minutes of waiting. That is fine for a nightly job. It is fatal for anything a person is staring at.

Record both meters on every attempt

The cheapest fix is a ledger. One line per attempt, written before you decide anything about the result. No SDK assumptions — if your transport exposes token counts, log them; if it does not, log characters and label the field as an estimate.

# runledger.py — one JSONL line per model attempt.
import json, time, pathlib, hashlib

LEDGER = pathlib.Path("runs/ledger.jsonl")
LEDGER.parent.mkdir(exist_ok=True)

def attempt(task_id, prompt, client, accept, **gen_kw):
    rec = {
        "task": task_id,
        "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()[:12],
        "prompt_chars": len(prompt),
    }
    started = time.monotonic()
    resp = client.generate(prompt, **gen_kw)      # your transport, free or paid
    gen_s = time.monotonic() - started
    ok = accept(resp)
    rec.update(
        gen_s=round(gen_s, 2),
        wall_s=round(time.monotonic() - started, 2),
        prompt_tokens=getattr(resp.usage, "prompt_tokens", None),
        completion_tokens=getattr(resp.usage, "completion_tokens", None),
        accepted=bool(ok),
    )
    with LEDGER.open("a") as fh:
        fh.write(json.dumps(rec) + "\n")
    return resp, ok

Note what is not in that record: the model output. You are logging cost, not content. Keep the outputs in your own store so the ledger stays small enough to read.

Let the cap fall out of the arithmetic

Most teams pick a retry cap by feel — three, five, whatever the last person typed. You can derive it instead. If each failure appends roughly e tokens to an initial context of c, the cumulative token cost of n attempts is n*c + e*n*(n-1)/2. Invert that against your ceiling.

def attempt_cap(context_tokens, error_tokens, ceiling_tokens):
    """How many attempts fit under the ceiling, including re-sends."""
    spent = 0
    n = 0
    while True:
        spent += context_tokens + n * error_tokens   # attempt n+1 re-sends n errors
        if spent > ceiling_tokens:
            return n
        n += 1

print(attempt_cap(context_tokens=8000, error_tokens=400, ceiling_tokens=120000))

Run that with your real context size and the answer is usually smaller than the number you were using. That is the whole point. The cap is a budget decision, not a personality trait.

Aggregate by accepted task, not by call

A per-call dashboard flatters itself: every call succeeded, therefore the system is efficient. Group by task and charge the failed attempts to the task, because that is where the money went.

import json, collections

runs = collections.defaultdict(lambda: {"tok": 0, "s": 0.0, "n": 0, "ok": False})
for line in open("runs/ledger.jsonl"):
    r = json.loads(line)
    g = runs[r["task"]]
    g["tok"] += (r["prompt_tokens"] or 0) + (r["completion_tokens"] or 0)
    g["s"] += r["wall_s"]
    g["n"] += 1
    g["ok"] = g["ok"] or r["accepted"]

done = [g for g in runs.values() if g["ok"]]
print("accepted:", len(done), "of", len(runs))
print("tokens/accepted:", sum(g["tok"] for g in done) // max(len(done), 1))
print("seconds/accepted:", round(sum(g["s"] for g in done) / max(len(done), 1), 1))
print("attempts/accepted:", round(sum(g["n"] for g in done) / max(len(done), 1), 2))

The third number is the one that tells you whether to raise your acceptance test or lower your cap. If attempts per accepted task drifts above two, your prompts are guessing, not working, and no amount of free capacity will fix a guessing loop.

When free capacity is the wrong bet

Free capacity is a fine trade when you are spending queue time instead of money. It is a bad trade the moment the queue time lands on someone else's critical path.

Job shape	Free capacity	Committed capacity
Someone is watching a spinner	rarely worth it	yes
Overnight batch, due by morning	yes, with a derived cap	only if the deadline is contractual
Long context with deep retries	only if the cap is enforced	yes, because each retry re-sends everything
Burst of one-off jobs	yes, accept the jitter	yes, if jitter breaks a downstream SLA

Read the second column carefully. It is not a downgrade. Trading seconds for tokens is a legitimate engineering choice — as long as you know which currency you spent and you wrote it down.

Limits of this method

This ledger measures only what your transport exposes. If token counts are missing, your totals are estimates and should be labelled that way. Queue wait varies by hour of day, so a single afternoon sample will mislead you; collect for at least a week before you set a cap in production. The numbers are yours and they do not transfer — another team's tokens-per-task tells you nothing about your prompts.

Do not use this approach if you have a hard external deadline, a sub-second interactive surface, or a compliance rule that keeps prompts inside your own network. And if your acceptance test is "looks fine to me," skip the ledger entirely; without a real test, accepted is a constant and the third metric becomes noise.

Where the free tier actually fits

There is a project worth knowing about here: MonkeyCode is an open-source coding agent project, and its operator states that new accounts receive a 10,000,000-token allowance on its free models along with a free server option. Treat that as operator-supplied and check the current project page before you plan a budget around it, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The honest way to use it is as the measurement environment, not the production answer. Point the ledger at free models, run your own task set for a week, and you get real tokens-per-accepted-task and seconds-per-accepted-task numbers without paying for the observation. The free server option means the collector and a cron for the aggregator can live somewhere that is not your laptop.

Then read your own table. If seconds per accepted task is fine for your workload, keep spending the cheap currency. If it is not, you now have a number to argue with instead of a feeling. If you want to try it, the free models and the free server are a low-cost place to start — and if you self-host the ledger instead, that works too, because the point was never the provider. The point was pricing the run.

Book the Partial Before You Retry

Quinn Li — Tue, 13 Oct 2026 02:03:13 +0000

A free model call that you abandon is not free. You already spent the queue, and you already spent the tokens the server emitted before you cut the socket. Write that partial off before you retry. If you do not, you are running a loop that looks thrifty and bills you in minutes.

You hit this when a shared free server is slow and your client is optimistic. The first attempt streams a couple hundred tokens, then the deadline you promised a person slips, so you cancel and send the same prompt again. The second attempt waits behind work you did not actually pull out of the pool by closing the socket. A zero price on the token meter does not rewind that queue.

It only hides the wait inside a dashboard that still says free. The retry question is live in developer discussions again. Treat it as a ledger problem, not as a mood your HTTP client is in.

Picture a print shop that takes night jobs at no charge. You submit a chapter, they set type for three pages, then you cancel because the pickup window moved. Those pages stay on the bench. Submitting the chapter again adds a second job behind the first.

What you book

Keep three numbers on the job, and update them after every attempt. spent_s is queue plus generation you already waited through. spent_tokens is what came back, including a partial you will not ship. ceiling_tokens is a cap you chose for this job, not a vendor quota copied out of a post.

The cap exists so a free lane cannot mean an unbounded loop inside your own process. A retry on that lane is allowed only when the next attempt can finish inside the remaining deadline, and only when spent tokens plus a fresh worst-case estimate still fit inside the ceiling. Otherwise you cut over to a paid lane for one clean attempt, or you stop and return what you have.

Stopping is a result. Another hopeful enqueue is not. Pass an empty attempt list for the first call, so the same function admits or refuses before you have a partial and you are not maintaining two policies.

The figures in the fixtures are placeholders you replace with timings you took this week. Nothing here is a measurement of a public pool, a price list, or a limit that will still be true next Monday. If a number is older than your last sample, it is already the wrong input.

A ledger you can run

Save the script as book_partial.py. It is a local decision helper, not a client for any vendor, and it will not open a socket. You feed it history. It returns an action.

#!/usr/bin/env python3
"""Write off partial free-lane attempts before you allow another retry."""

from __future__ import annotations

import json
import sys
from dataclasses import dataclass


@dataclass(frozen=True)
class Attempt:
    queue_s: float
    gen_s: float
    emitted_tokens: int
    shipped: bool


@dataclass(frozen=True)
class Policy:
    deadline_s: float
    ceiling_tokens: int
    next_queue_s: float
    next_gen_s: float
    next_est_tokens: int
    paid_queue_s: float
    paid_usd_per_1k: float


def book(attempts: list[Attempt], policy: Policy) -> dict:
    spent_s = sum(a.queue_s + a.gen_s for a in attempts)
    spent_tokens = sum(a.emitted_tokens for a in attempts)
    remaining_s = policy.deadline_s - spent_s
    next_free_s = policy.next_queue_s + policy.next_gen_s
    next_paid_s = policy.paid_queue_s + policy.next_gen_s
    projected = spent_tokens + policy.next_est_tokens

    if any(a.shipped for a in attempts):
        action, why = "stop", "a previous attempt already shipped"
    elif projected > policy.ceiling_tokens:
        action, why = "stop", "another try would cross your token ceiling"
    elif next_free_s <= remaining_s:
        action, why = "retry_free", "remaining clock covers one more free attempt"
    elif next_paid_s <= remaining_s:
        action, why = "cut_over", "free retry does not fit; one paid attempt does"
    else:
        action, why = "stop", "no lane finishes inside the time you have left"

    sunk_usd = (spent_tokens / 1000.0) * policy.paid_usd_per_1k
    return {
        "action": action,
        "why": why,
        "spent_s": round(spent_s, 2),
        "remaining_s": round(remaining_s, 2),
        "spent_tokens": spent_tokens,
        "projected_tokens": projected,
        "sunk_usd_if_those_tokens_were_paid": round(sunk_usd, 4),
    }


def main() -> int:
    raw = json.load(sys.stdin)
    attempts = [Attempt(**item) for item in raw["attempts"]]
    policy = Policy(**raw["policy"])
    json.dump(book(attempts, policy), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The first fixture is a night job with a wide clock. One partial came back, nothing has shipped, and you still have room under a ceiling you set yourself.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

You should see retry_free. Spent time is 44 seconds, the remaining window is wide, and 220 plus a guessed 800 stays under 4000. The dollar field is a shadow price so the partial stays visible on the record. It is not an invoice, and the rate is a stand-in you must replace.

If you do not know a current paid rate, set paid_usd_per_1k to 0 and decide on time and tokens alone. A stale price copied from an old thread is worse than a zero, because it looks like evidence.

Now the interactive case. Same partial, a 60 second deadline, and the same next guess. The night job and the inline job are not the same bet, even when they share a prompt.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 60,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

Remaining time is 16 seconds. The next free attempt wants 47. One paid attempt wants 14, which still fits, so the action is cut_over. The free server did not become a bad product in that branch; the remaining clock stopped matching the queue you would sit in again.

If you lower the ceiling under the projection, the same function stops instead of retrying, even when the deadline is wide. Run the third fixture and read the action before you argue with it.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 500,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

Projected tokens are 1020 against a ceiling of 500, so you stop. A long deadline does not authorize an open token loop just because the lane advertises free model access. Your ceiling is how you keep that advertisement from becoming a retry policy. Count tokens the same way on every attempt, or the stop branch fires late and you will blame the wrong line.

Pin the branches

Wire a tiny assertion so the three branches cannot drift while you edit the formula. This is a local test. It does not touch a network, and it fails if you reorder the checks without updating the expected actions.

python3 - <<'PY'
import importlib.util

spec = importlib.util.spec_from_file_location("book_partial", "book_partial.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)

def run(deadline, ceiling, emitted=220):
    attempts = [mod.Attempt(35, 9, emitted, False)]
    policy = mod.Policy(deadline, ceiling, 35, 12, 800, 2, 0.002)
    return mod.book(attempts, policy)["action"]

assert run(600, 4000) == "retry_free"
assert run(60, 4000) == "cut_over"
assert run(600, 500) == "stop"
assert run(60, 500) == "stop"
print("ok")
PY

Run that from the directory where you saved the file, then read the last assert with the cut-over case beside it. A tight deadline and a blown ceiling both stop, and stop wins over cut-over because the ceiling check runs first. If you would rather pay once the free path has already burned a partial, swap those branches and re-run the test until it matches the policy you actually want. A client that retries on its own will not honor either order, so turn that off or the ledger is a story you tell after the loop has already run.

Sample the queue you will sit in

Before you trust next_queue_s, take a sample in the hour you actually run. Send a short fixed prompt on the free lane, record time to first token as queue, and record the rest as generation. Do that three times, keep the worst queue, and discard the probe text. If the probe fails, the lane is not eligible this hour.

Do not retry the probe until it feels fine. That loop is the one the ledger is meant to stop. The command below is a stand-in timer, not a benchmark and not a vendor call. Replace the sleep with your own client, then put the worst sample into the policy.

python3 - <<'PY'
import time

start = time.perf_counter()
# Stand-in only. Replace this sleep with one short probe call.
time.sleep(0.05)
queue_s = time.perf_counter() - start
print(f"sample_queue_s={queue_s:.3f}")
PY

Where a free lane still belongs

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode belongs in this workflow only as a spill lane: free model access and a free server option, as supplied for this note, for jobs the ledger has already marked retry_free. This page does not name models, token grants, hardware, or how long either option lasts, because those facts move and a post should not freeze them. If you still want that spill lane, read the live limits, copy the ones you will depend on into ceiling_tokens and next_queue_s, and run the assertion again when the sample changes.

Free capacity is the wrong bet when a person is blocked on the answer, when the attempt you would abandon is not idempotent, or when you cannot sample queue time and would be typing a wish into next_queue_s. It is also the wrong bet when a retry can double-apply a side effect, since booking tokens will not undo a second write. Stamp your own idempotency key on the job before the first call. Treat shipped: true as final, so a late free response cannot overwrite a paid answer you already returned.

What this leaves out

The function trusts your samples. One quiet afternoon is not a week, so if the queue swings, keep the worst sample from the hour you run and put that number in next_queue_s. It also treats every emitted token as spent, which is the right bias for a ceiling and the wrong bias for a quality score. A partial you can resume is not the same shape as a partial you must throw away.

If your client supports continuation, add only the missing gap to the projection, then re-run the assertion so the new bias is the one you tested. Do not use this for payments, access control, or any path where stop is not an acceptable outcome and you have no contracted fallback. Do not use it as a capacity forecast for a team. It books one job's history, and it will not tell you whether a free server can hold everyone at the top of the hour.

You will still be tempted to skip the write-off, because the dashboard said the call was free and the partial looked small. Book it anyway. The next retry is where a free lane turns into a queue you cannot see the end of.

Write the Prefix Down Before the Slot Dies

Quinn Li — Sun, 11 Oct 2026 21:51:35 +0000

A preempted stream is not a free retry. If the lane already emitted tokens, those tokens are spent, and starting the prompt over buys a prefix you already hold.

You feel the urge anyway. The socket drops, the free slot vanishes, and the editor still shows an unfinished function. The cheap instinct is to send the same messages again and hope the next slot is kinder. That instinct turns a short outage into a second bill for the same answer.

Think of a courier who already walked the parcel to your door, then got pulled off the route. You do not hire a second courier to repeat the walk from the warehouse. You keep what arrived and pay only for the last flight of stairs.

A model stream has that shape, with one wrinkle. Input context may have to be shown again, because a provider often will not hand you a receipt for cached prompt tokens. Output you already received must not be generated again. Those are different costs, and mixing them up is how the bill grows while the answer stays the same.

This note is a small ops habit for that split. It is a proposal, not a measured result, and the sample is unexecuted example code. It does not assume a model name, a quota, a machine size, or a promise that a free lane will exist next week.

Before the call, write a job row. Give it an id, a hash of the prompt, a token budget, and a cursor at zero. The hash is how you refuse to resume against a prompt you have since edited. If the hash does not match, the cursor belongs to a different question, and stitching it on would answer the wrong ask.

While tokens arrive, flush the cursor on a short interval, not only at the end. A flush at the end is a diary of success. A flush every few dozen tokens is a diary of the failure you actually have.

Store the assistant text, the count you treat as spent, and a monotonic revision. If the process dies between flushes, you lose only the gap. That gap is the acceptable loss. The prefix before it is not.

On the way back in, do not rebuild the original request and ask for a fresh completion. Build a continuation, and cap output at budget minus spent. If the API allows an assistant prefill, put the stored prefix there and ask it to continue without repeating itself.

If the API will not accept a prefill, send a short instruction plus the unfinished tail, not the entire original thread plus a demand for a new essay. If the remainder is below a floor you choose, abandon. A continuation that may spend the original budget again is just a replay wearing a new function name.

Here is the gate in plain Python. Save it as budget_cursor.py. Treat every number as a knob, not a measurement. The annotations assume a current Python 3.

import hashlib, json
from pathlib import Path

ROOT = Path(".jobstate")

def prompt_hash(messages: list[dict]) -> str:
    blob = json.dumps(messages, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()

def open_job(job_id: str, messages: list[dict], budget_tokens: int) -> dict:
    ROOT.mkdir(exist_ok=True)
    path = ROOT / f"{job_id}.json"
    if path.exists():
        job = json.loads(path.read_text())
        if job["prompt_hash"] != prompt_hash(messages):
            raise SystemExit("prompt changed; refuse stale cursor")
        return job
    job = {
        "job_id": job_id,
        "prompt_hash": prompt_hash(messages),
        "budget_tokens": budget_tokens,
        "spent_tokens": 0,
        "prefix": "",
        "revision": 0,
        "state": "open",
    }
    path.write_text(json.dumps(job))
    return job

def flush(job: dict, piece: str, piece_tokens: int) -> dict:
    job["prefix"] += piece
    job["spent_tokens"] += piece_tokens
    job["revision"] += 1
    if job["spent_tokens"] > job["budget_tokens"]:
        job["state"] = "over_budget"
    path = ROOT / f"{job['job_id']}.json"
    tmp = path.with_suffix(".json.tmp")
    tmp.write_text(json.dumps(job))
    tmp.replace(path)
    return job

def continuation(job: dict, floor: int = 32) -> dict | None:
    remain = job["budget_tokens"] - job["spent_tokens"]
    if job["state"] != "open" or remain < floor or not job["prefix"]:
        return None
    return {
        "assistant_prefill": job["prefix"],
        "max_output_tokens": remain,
        "instruction": "Continue from the prefill. Do not repeat it.",
    }

The temp-file rename avoids a half-written JSON line if the process dies mid-flush on the same filesystem. It is not a distributed lock. Two workers sharing one job id will still clobber the cursor, so give each in-flight call its own id.

You can exercise the refusal path with no network and no key. The 12 below is a stand-in, not a tokenizer result. In a real call you store the usage figure for that chunk, not a guess from string length.

mkdir -p .jobstate
python - <<'PY'
from budget_cursor import open_job, flush, continuation
msgs = [{"role": "user", "content": "draft a migration note"}]
job = open_job("mig-14", msgs, budget_tokens=400)
flush(job, "Step 1. Freeze writes.\n", 12)
print(continuation(job))
print("remain", job["budget_tokens"] - job["spent_tokens"])
PY

If continuation returns None, you stop. You do not send the original call once more just in case. None means the prefix already consumed the useful budget, or there is no prefix worth resuming. A full replay would ignore the ending you just computed.

Watch the file if you want to see the cursor move under a fake stream. This loop is still local. It proves the flush, not a provider.

python - <<'PY'
import json
from budget_cursor import open_job, flush
job = open_job("mig-14", [{"role": "user", "content": "draft a migration note"}], 400)
for piece, n in [("Step 2. Copy rows.\n", 10), ("Step 3. Verify counts.\n", 11)]:
    flush(job, piece, n)
print(json.dumps(job, indent=2))
PY

You have three exits, and you should pick one before the call, not during the panic. Abandon when the remainder is under the floor or the prefix is empty. Resume when the prefix is real and the remainder still fits the budget.

Spill to a lane you are willing to wait on only when the job cannot be dropped and the free slot has already failed. Spilling is not a loop. One spill, then a human, or a queue that can actually reserve the work. The reservation problem is older than this cursor, and this note does not solve it.

A free lane is a decent stage for that rehearsal, and a bad home for work you cannot drop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The useful fit is narrow: MonkeyCode's free model access and a free server option can host a broken-stream drill, after you read the live limits in the product you have, and only for jobs you can abandon. This note does not record a quota, a card, a region, or a duration, because those figures move and no primary source for them was attached.

Free capacity is the wrong bet when the remainder is not optional. A migration note that can be rewritten later may die at a cursor. A payment capture, a mailbox send, an infrastructure apply, or a customer-visible webhook may not. Those need an idempotency key, and the side effect should run after the text is finished, not inside the stream. If a partial answer is worse than silence, do not emit it onto a lane that can vanish mid-sentence.

There is a quieter wrong bet than preemption: input replay. If the provider bills the prompt again on every continuation, a huge system prompt will swallow the savings from not regenerating the prefix. Log input tokens and output tokens separately on the resume call. If resumed input is nearly the original prompt plus the stored prefix, you moved the cost, you did not remove it.

Shrink the resumed context to the tail you still need, or stop using continuation for that prompt shape. The cursor still earns its keep when it blocks a second full completion. It will not make a bloated prompt cheap, and it will not refund tokens a vendor has already metered.

Skip this habit if you cannot persist the prefix outside the process. A cursor that lives only in memory dies with the preempt and leaves the replay you were trying to avoid. Skip it when the client hides token counts and you would be guessing piece_tokens. A guessed cursor is a lie in the ledger.

Skip it when the attempt must be byte-identical to the first sample. Continuation is a new draw, not a restored sampler, unless the provider documents otherwise. Do not claim determinism you did not buy. And skip it for any call that mutates the world before the stream ends, because a resume would be a second chance to mutate it.

The habit is small enough to keep. Open a job, flush while text arrives, and on death ask only for the remainder or ask for nothing. A free slot can host the drill. It cannot refund a prefix you already spent, and it cannot be the commitment under work you are unwilling to lose.

Free Tokens Do Not Pay the Clock

Quinn Li — Sat, 10 Oct 2026 18:28:01 +0000

Free tokens do not pay a deadline. If the seat is slow, each retry spends minutes you cannot refund, and the next call still has to fit inside what is left.

You feel the trap when a repair loop looks harmless. The prompt failed once, the seat is marked free, and another attempt costs nothing on the rate card. Then the request sits. The next attempt starts late, sometimes after the caller has already given up.

You did not buy extra intelligence. You bought a slower no. Treat it like a restaurant that comps the meal and still holds the table for an hour. The food is free, and the evening is not.

Model work splits the same way. A token grant can zero the model line while the clock keeps charging the humans and jobs waiting on the result. Cost work here is four coupled numbers, not a vibe about cheap inference.

What you are actually budgeting

Before you attach a retry policy to any free seat, write down four numbers you can measure without a vendor dashboard. You need a deadline, a token ceiling for the whole job, a maximum number of attempts, and the longest queue wait you will tolerate. If you cannot name those four, you do not have a policy. You have a hope.

The deadline is wall-clock, not a feeling. Pick a duration the caller can still use, such as 45 seconds for an interactive repair or 10 minutes for a batch note. The token ceiling is the most you will let one job read and write, prompt plus completion, across every attempt.

The attempt cap stops a flaky case from eating the evening. The wait cap is the number people skip, and it is the number that makes a free seat expensive. A call that is free on the rate card and late on the clock has already failed the job you claimed to care about.

This ledger is a proposal, not a trace from a production run. It does not call a model. It decides whether another attempt is still admissible given the time and tokens you have left. Run it locally, and label the output as a sketch, before you point any client at a live seat.

from dataclasses import dataclass

@dataclass
class Budget:
    deadline_s: float
    token_ceiling: int
    max_attempts: int
    max_wait_s: float

@dataclass
class Attempt:
    wait_s: float
    elapsed_s: float
    tokens: int

def admit(budget, spent_s, spent_tokens, attempts_used, nxt):
    if attempts_used >= budget.max_attempts:
        return 'refuse: attempt cap'
    if nxt.wait_s > budget.max_wait_s:
        return 'refuse: queue wait'
    remaining = budget.deadline_s - spent_s
    if nxt.wait_s + nxt.elapsed_s > remaining:
        return 'refuse: deadline'
    if spent_tokens + nxt.tokens > budget.token_ceiling:
        return 'refuse: token ceiling'
    return 'admit'

if __name__ == '__main__':
    budget = Budget(deadline_s=45, token_ceiling=8000, max_attempts=2, max_wait_s=5)
    # Unexecuted sketch. Replace the numbers with timings you actually observed.
    rows = [
        (0.0, 0, 0, Attempt(1.2, 6.0, 2100)),
        (7.2, 2100, 1, Attempt(18.0, 9.0, 2400)),
    ]
    for spent_s, spent_tokens, used, nxt in rows:
        print(admit(budget, spent_s, spent_tokens, used, nxt))

The first sketch attempt fits. The second does not. An 18 second queue already blows a 5 second wait cap, and the remaining clock cannot honestly host another full call. That refusal is the result you wanted.

You stop before a complimentary seat turns a short repair into a late one. Walk the same numbers so the function is not a black box. Your caller can wait 45 seconds. Scratch attempt one waits 1.2 seconds and finishes in 6, using about 2100 tokens.

You then have 37.8 seconds and 5900 tokens left, with one attempt remaining. Scratch attempt two sits 18 seconds before the first byte. Even if the model answered instantly, you have already spent more wait than the cap. Admitting it would teach the client that free means late.

Refuse, log the wait, and change the seat or the prompt. You can do the same check on paper, without the script. Remaining seconds must exceed observed wait plus a conservative elapsed time from the last scratch run that actually returned. Remaining tokens must exceed the last prompt size plus the last completion size, and attempts used must sit under the cap.

All three have to pass. One failure refuses the call. That is stricter than a typical retry helper, which often looks only at the HTTP status and then sends the transcript again. Status is not a budget.

How to rehearse it without spending a real job

You can collect the inputs with a stopwatch and a token estimate. Start a scratch prompt you are willing to throw away. Record queue wait as the gap between submit and first byte. Record elapsed time through the last byte.

Record tokens from the response usage field when the API returns one, or from a local tokenizer when it does not. Write those three numbers down beside the decision. Do not average a week of runs into a single comfortable mean. A mean hides the wait that will miss the deadline.

Time the gap in the shell so the log has a clock, not a memory. This snippet only measures pacing you already observed. It does not call a provider, and the offsets are stand-ins until you paste real timestamps.

python3 - <<'PY'
import time
start = time.perf_counter()
submitted = time.perf_counter()
first_byte = submitted + 1.2
done = first_byte + 6.0
print(f'wait_s={first_byte - submitted:.3f} elapsed_s={done - submitted:.3f}')
print(f'rehearsal_wall_s={time.perf_counter() - start:.3f}')
PY

Append each observed row before you raise the attempt cap. Keep the decision next to the inputs so a later reader can see why the gate closed.

python3 ledger.py | tee -a rehearsal.log
printf '%s\n' 'wait_s=18.0 elapsed_s=9.0 tokens=2400 decision=refuse: queue wait' >> rehearsal.log

Read the log cold, a day later if you can. If two scratch runs already show waits near the cap, a third automatic retry is not a tuning idea. It is a plan to miss the deadline on purpose. Fix the prompt, shrink the context, or move the job to a seat you can hold.

There is a second bill hiding in the retry itself. Each new attempt often resends the same transcript. A 2100 token prompt, sent three times, is not 2100 tokens of work. It is 6300, plus three completions, plus three trips through the queue.

A free grant makes that multiplication feel fake. It is not fake. Grants end, and a duplicated transcript is a fast way to spend one on copies of a prompt you already paid for in time. Cap tokens per job, not only per pretty-looking call.

A timeout caused by the queue will tempt the client to send again while the first request is still in line. Now two of your calls are waiting. You doubled the wait, and you may double the tokens if both complete. Keep one in-flight attempt per job until a scratch log shows the seat keeping up.

Do not put keys, customer text, or private logs into the scratch prompt or into rehearsal.log. The gate only needs wait, elapsed time, and a token count. If a row needs a secret to make sense, the rehearsal is pointed at the wrong example.

Where a free seat fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those are useful as a scratch seat for the rehearsal above: you can price wait, tokens, and retries on a call you are allowed to discard.

This note does not claim a quota, a model list, a hardware shape, a duration, or a benchmark. If those details matter to your job, read the current project docs and treat them as time-sensitive. A number you remember from a launch post is not a control you can schedule against.

Use that scratch seat when the work is disposable. A prompt you are shaping, a parser you are testing, a failure you want to see once: that is the fit. The free option earns its place because a thrown-away call should not land on a paid invoice, and because you want queue behavior in the log before you trust it.

You are buying information about the seat, not a result you mean to ship. Once the log says the wait is inside the cap, you still re-check on the seat you will actually use for the real job. A good scratch hour does not transfer.

Do not use a complimentary seat for a committed job. A customer request with a latency promise, a migration step that must finish inside a window, or a nightly close that pages someone if it slips needs a seat you can hold, or a deadline you can move. Leftover room is not a reservation.

You do not get to assume the room will be empty when the retry fires. If the first observed wait already exceeds max_wait_s, refuse the next attempt on that seat. Hoping the queue clears is not a control. Neither is refreshing a dashboard until the line looks short.

The same rule applies when the token line is still at zero and the clock is not. A repair that looks free can still miss a human waiting on a review, a CI job holding a runner, or a teammate who planned to leave. Price those minutes as real cost. If the admit function would refuse, the label on the seat does not overrule it.

Who should skip this

Skip the ledger if you have no deadline and no shared seat. A solo notebook experiment with nobody waiting does not need admission control. Also skip it if you cannot observe wait time, because a token-only policy will green-light a call that arrives after the user has left.

Do not use this sketch as a scheduler for production traffic. It is a local gate for one job's next attempt. It does not know other tenants, it does not retry with backoff magic, and it will not save you from a provider outage. If you need fairness across a fleet, this file is the wrong tool.

There is a quieter failure mode. You can tune the caps until every historical row is admitted, then call the policy safe. That is curve-fitting, not ops. Keep one held-out scratch run that you did not use to pick the numbers.

If that run refuses, believe the refusal. If you cannot explain the refusal in one sentence a teammate understands, the cap is decoration. Raise the deadline in public, or shrink the job, rather than silently widening the gate until it never says no.

If you already keep a scratch budget and a job you can afford to delay, rehearse the gate on MonkeyCode's free model access and free server before you attach the same policy to anything with a pager. The log, not the word free, tells you whether the seat is cheap enough to keep.

Stamp a Deadline Before You Pick a Lane

Quinn Li — Fri, 09 Oct 2026 14:57:14 +0000

A zero token price does not make a job free. If the work has a deadline, the queue wait and the retry fan-out are the invoice, and a free lane is the wrong place to send it.

You already know the cheap thrill: a prompt returns, the dashboard shows nothing owed, and the next attempt feels harmless. Then the job sits, a worker restarts, and the same prompt goes out again because nobody stamped a stop. The token counter stays polite. The clock does not.

Treat that gap as an admission problem, not a model problem. Before you pick a lane, write down the deadline, the worst wait you will tolerate, and how many attempts you will allow. If those three numbers cannot coexist, do not send the job. A later, smarter model will not repair a queue you never bounded.

Picture a ferry that boards for nothing and a taxi that prints a fare. The ferry is fine when you are carrying a sample and you can miss the evening, and it is a bad vehicle when someone is waiting on the other shore and the boat has no posted departure. A zero-price lane works like that ferry, while a lane you can actually hold works like the taxi. You match the vehicle to the promise you already made, not to whichever ride looks cheaper.

What you are actually buying

Time, tokens, and retries are one budget wearing three names. Queue time is time you do not control. Generation time is time you half control. Tokens are the residue of both, because a retry sends the prompt again and often sends a longer context than the first try.

A free price zeroes only the third name. It does not zero the first two. If the machine you did not hold is busy, preempted, or simply slow to schedule, you can miss a deadline while spending no money at all. That is still a failed job, because your caller does not grade you on the invoice.

Retries make the math worse. Two attempts are not twice as innocent as one when each attempt waits in line. Four attempts with a cold start can cost more wall clock than one attempt on a lane you held, even when the held lane has a nonzero token rate. Compute that comparison before you feel loyal to the zero.

This is also how a casual rerun becomes an incident. The second run is not a copy of the first. It is a new wait, a new context, and sometimes a new failure mode. If you cannot point at a hold, or at an explicit decision that this job may miss its window, you are guessing with someone else's afternoon.

A smaller model can still be the right model. That choice is about output quality per token, and it belongs in a different column from lane admission. A cheap model that starts immediately can beat a free model that sits. Do not collapse those columns, or you will celebrate a price you never paid while the queue eats the window.

A ledger you can run locally

The snippet below is a proposal. It is not a benchmark, and it was not executed against a live provider for this article. It only encodes the rule: deadline work does not ride an unreserved lane, and the retry fan-out must fit inside the window.

from dataclasses import dataclass

@dataclass(frozen=True)
class Attempt:
    lane: str
    queue_wait_s: float
    gen_s: float
    prompt_tokens: int
    completion_tokens: int
    succeeded: bool

def summarize(attempts, prompt_rate, completion_rate):
    tokens = 0
    wall = 0.0
    money = 0.0
    for a in attempts:
        tokens += a.prompt_tokens + a.completion_tokens
        wall += a.queue_wait_s + a.gen_s
        if a.lane != "free":
            money += a.prompt_tokens * prompt_rate
            money += a.completion_tokens * completion_rate
    return {"tokens": tokens, "wall_s": wall, "money": money}

def admit(deadline_s, predicted_wait_s, max_attempts, predicted_gen_s, lane):
    if deadline_s is not None and lane == "free":
        return False, "deadline jobs do not ride the free queue"
    worst = max_attempts * (predicted_wait_s + predicted_gen_s)
    if deadline_s is not None and worst > deadline_s:
        return False, "retry fan-out breaks the deadline"
    if max_attempts < 1:
        return False, "nothing to run"
    return True, "ok"

Save it as lane_admit.py. Then check the branches with the standard library. No network, no key, and no account are required.

python - <<'PY'
from lane_admit import Attempt, summarize, admit

probe = [
    Attempt(lane="free", queue_wait_s=40, gen_s=6,
            prompt_tokens=900, completion_tokens=180, succeeded=False),
    Attempt(lane="free", queue_wait_s=55, gen_s=7,
            prompt_tokens=900, completion_tokens=40, succeeded=False),
]
held = [
    Attempt(lane="reserved", queue_wait_s=1.5, gen_s=5,
            prompt_tokens=900, completion_tokens=180, succeeded=True),
]
assert summarize(probe, 0.000002, 0.000008)["money"] == 0.0
assert summarize(probe, 0.000002, 0.000008)["wall_s"] > summarize(held, 0.000002, 0.000008)["wall_s"]
assert admit(30, 40, 2, 6, "free")[0] is False
assert admit(12, 8, 2, 5, "reserved")[0] is False
assert admit(None, 40, 1, 6, "free")[0] is True
print("lane checks passed")
PY

Read the three decisions in order. The free lane is refused because a deadline was set, and the held lane is refused too, in this made-up example, because two attempts still overflow twelve seconds. Only the last call is admitted, because you passed deadline_s=None, and that call is a probe. It may wait or be dropped, but it must not sit on the path that pages a human.

The rates in the command are placeholders so the arithmetic is visible. Do not treat them as a vendor price. Swap in the figures from your current billing page when you care about money. When the lane is free, the function already forces the money term to zero, so you can still lose on wall_s.

Keep the inputs honest. predicted_wait_s should be a high percentile from your own recent attempts, not a number you liked in a demo. If you have no samples, pass a wait you are willing to be wrong about, and treat an admit as provisional. A function cannot discover a queue it has never seen.

Failed rows still count

You will want to delete the failures before you summarize. Don't, because a timeout still occupied a slot and still attached every prompt token you sent. If the server returned a partial completion, those tokens belong in the row. Drop them and the two charts disagree, so you fix the wrong one.

When you debug a slow evening, sort by queue_wait_s before you sort by answer length. A long answer is a generation problem, and a long wait is a lane problem. Mixing them is how a team swaps models for a week and never touches scheduling. The log has to separate those, because a dashboard that reports no token spend will not.

python - <<'PY'
import json
rows = [json.loads(line) for line in open("attempts.jsonl")]
rows.sort(key=lambda r: r["queue_wait_s"], reverse=True)
for r in rows[:5]:
    print(r["job_id"], r["lane"], r["queue_wait_s"], r["succeeded"])
PY

Give each attempt an idempotency key before it leaves your process. A retry must be the same job, not a new job with a fresh context and a second copy of the prompt. Without that key, this ledger under-counts, and your notebook will disagree with the platform about what you owe in time. Store the key, the lane, the wait, the token estimates, and whether you intended a deadline.

Run the zero-price path on purpose, as a bounded sample, not as the default route. Cap it at a count you can read, such as one hundred attempts that carry no deadline, and record wait, generation time, and whether the call returned. After that sample, put a high percentile wait into predicted_wait_s. If that percentile already exceeds the tightest deadline you owe a caller, that lane is disqualified for that caller no matter what the token price says.

Where a zero price earns a place

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode, as the operator described it for this note, includes a free model path and a free server option. Those are availability claims, not a measured quota, a hardware class, or a promise that the queue is short. Use them for the probe half of the workflow: prompt diffs, parser checks, and replaying recorded waits through admit.

Keep caller-visible work, deploy gates, and anything you will retry inside a window on a lane you can hold, or shed the job in your own code. Do not paste a token ceiling into the runbook from memory. Limits, model names, and windows move, and a stale figure is worse than a blank. Read the project docs on the day you enroll, write down what you observed, and attach that note to the job id.

If the docs are silent, record the free lane as zero price and unknown wait. Unknown wait is not a hold. If your attempt logs already exist, replay a week of them through admit before you move a user-facing job onto that free path. That replay is the only next step this note asks for.

What this will not do

The function does not know why a queue grew. It will not see a provider incident, a bad prompt, or a client that resubmits on every non-200. You still own the stop condition. A refusal inside admit is only useful if the caller actually stops.

It will not turn yesterday's samples into today's capacity. A quiet Tuesday does not hold Wednesday. If predicted_wait_s is a wish, the function will approve a lane you merely labeled reserved. Feed it measurements, or feed it a ceiling you can afford to miss.

Skip this approach if you need a contractual latency target and you have no lane you can hold. A refusal cannot manufacture capacity. Skip it if your team will treat a false result as advice and send the job anyway. Skip it, too, if the plan is to drop secrets, customer text, or production identifiers into a free probe because the token price is zero.

A free process is still a process you must account for, so check the current docs before you decide what it is allowed to see. Label every number you have not measured. The waits in the sample command are illustrations, picked so the branch is obvious, and your queue will not match them. If you publish a chart from this ledger, keep estimated tokens apart from invoiced tokens, or the fiction will win the meeting.

Stamp the deadline first. If the wait and the retries cannot fit inside that stamp, do not pick a lane. Learn the wait on the free path, and hold a lane or shed the job when a person is actually waiting on you.

Refuse the Round You Have Not Sent

Quinn Li — Thu, 08 Oct 2026 10:29:43 +0000

You do not control spend when the invoice shows up. You control it in the quiet second before the next model round leaves your process. A meter that has not moved is not permission to widen the loop. If that lane is free, the round still costs time, context, and the tool you are about to fire.

Picture a kitchen that has already called the next plate. The check in your hand describes the last plate, not the one on the grill. An agent run behaves the same way. The person typed one message. Behind it sits a chain of rounds: prompt, completion, tool result, prompt again.

That chain is the unit you should fence. A user turn is a poor proxy, because one turn can hide several model calls, and the later prompt is carrying tool text it did not start with. The round tends to get heavier, not lighter. You feel this when a short lookup starts pasting file bodies back into the prompt, and the next answer is mostly a recap of what the tool just returned.

The mistake is to treat a quiet meter as slack. Free model access feels like an empty pan, so you add another tool call, then another, because the dollar field stayed at zero. Free capacity is the wrong bet when the thing you are buying with it is a wider loop. Your caller still waits. Your tools still run.

Fence the unsent round

When the run starts, write down a ceiling in tokens and in milliseconds. After each completed round, keep the observation: prompt tokens, completion tokens, elapsed milliseconds, and the size of any tool payload you are about to attach. Before you send the next round, project it from the last observation plus that payload, and add the projection to what you have already spent. If the sum crosses the ceiling, return the partial result. Do not send the call and hope the other side is gentle.

The module below is a proposal you can run on your own machine. It is not a provider client, and it is not a bill. This article does not include a live model run. Token fields are observations you pass in. The byte divisor is a stand-in for a moment when you do not have that model's tokenizer, not a rule any vendor promised you.

"""round_fence.py

Proposal: refuse the next model round before you send it.
Counts are caller-supplied observations, not a provider invoice.
"""

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Ceiling:
    max_tokens: int
    max_millis: int


@dataclass(frozen=True)
class Observation:
    prompt_tokens: int
    completion_tokens: int
    elapsed_millis: int
    tool_result_bytes: int = 0


@dataclass(frozen=True)
class Decision:
    allow: bool
    reason: str
    projected_tokens: int
    projected_millis: int


def project_next(last, pending_tool_bytes, bytes_per_token=4):
    if bytes_per_token <= 0:
        raise ValueError("bytes_per_token must be positive")
    if pending_tool_bytes < 0:
        raise ValueError("pending_tool_bytes must be non-negative")
    extra = (pending_tool_bytes + bytes_per_token - 1) // bytes_per_token
    if last is None:
        return extra, 0
    prompt = last.prompt_tokens + extra
    return prompt + last.completion_tokens, last.elapsed_millis


def admit(
    ceiling,
    spent_tokens,
    spent_millis,
    last,
    pending_tool_bytes,
    bytes_per_token=4,
):
    projected_tokens, projected_millis = project_next(
        last, pending_tool_bytes, bytes_per_token
    )
    next_tokens = spent_tokens + projected_tokens
    next_millis = spent_millis + projected_millis
    if last is None:
        return Decision(
            False,
            "no observation yet; calibrate before you loop",
            next_tokens,
            next_millis,
        )
    if next_tokens > ceiling.max_tokens:
        return Decision(
            False,
            "next round would cross the token ceiling",
            next_tokens,
            next_millis,
        )
    if next_millis > ceiling.max_millis:
        return Decision(
            False,
            "next round would cross the time ceiling",
            next_tokens,
            next_millis,
        )
    return Decision(True, "within ceiling", next_tokens, next_millis)


def drive(observations, ceiling, bytes_per_token=4):
    """First item is a calibration send. Later items must pass admit().

    For a later item, tool_result_bytes is the payload you would attach
    before that send, not a bill for a call you already made.
    """
    spent_tokens = 0
    spent_millis = 0
    last = None
    sent = []
    for index, obs in enumerate(observations):
        if last is not None:
            decision = admit(
                ceiling,
                spent_tokens,
                spent_millis,
                last,
                obs.tool_result_bytes,
                bytes_per_token,
            )
            if not decision.allow:
                return {
                    "sent": sent,
                    "refused_index": index,
                    "reason": decision.reason,
                    "projected_tokens": decision.projected_tokens,
                    "projected_millis": decision.projected_millis,
                }
        spent_tokens += obs.prompt_tokens + obs.completion_tokens
        spent_millis += obs.elapsed_millis
        last = obs
        sent.append(index)
    return {
        "sent": sent,
        "refused_index": None,
        "reason": "loop exhausted",
        "projected_tokens": spent_tokens,
        "projected_millis": spent_millis,
    }

Read admit and drive as two different doors. admit will not bless a cold start, because an elapsed time of zero is not a measurement. drive lets the first observation through so you have something to project from, then asks admit before every later round. That first send is a hole. Completion length is unknown until it comes back, so the first round can overshoot a ceiling you only enforce from round two. If you cannot afford one unbounded completion, do not start. Cap the prompt you already know, and keep the first tool read-only.

What the local tests lock

The tests use fake observations. They do not call a network. They lock the refusal shape: a heavier tool payload trips the token ceiling, a slow observation trips the time ceiling even when tokens would fit, a short loop inside both limits is allowed to finish, and a cold admit refuses.

"""test_round_fence.py — local proposal, no network."""

from round_fence import Ceiling, Observation, admit, drive


def test_refuses_when_next_round_crosses_token_ceiling():
    ceiling = Ceiling(max_tokens=100, max_millis=10_000)
    rounds = [
        Observation(40, 10, 200, tool_result_bytes=0),
        Observation(40, 10, 200, tool_result_bytes=80),
    ]
    result = drive(rounds, ceiling)
    assert result["refused_index"] == 1
    assert result["sent"] == [0]
    assert "token ceiling" in result["reason"]
    # spent 50; next prompt 40 + ceil(80/4) plus last completion 10 => 70
    # 50 + 70 = 120, which is the projected total on the refusal
    assert result["projected_tokens"] == 120


def test_refuses_on_time_even_if_tokens_fit():
    ceiling = Ceiling(max_tokens=10_000, max_millis=500)
    rounds = [
        Observation(10, 5, 400, 0),
        Observation(10, 5, 400, 0),
    ]
    result = drive(rounds, ceiling)
    assert result["refused_index"] == 1
    assert "time ceiling" in result["reason"]
    assert result["projected_millis"] == 800


def test_allows_a_short_loop_inside_both_ceilings():
    ceiling = Ceiling(max_tokens=500, max_millis=5_000)
    rounds = [
        Observation(20, 10, 100, 0),
        Observation(20, 10, 100, 16),
    ]
    result = drive(rounds, ceiling)
    assert result["refused_index"] is None
    assert result["sent"] == [0, 1]


def test_cold_start_is_not_a_permit():
    decision = admit(Ceiling(100, 1000), 0, 0, None, 40)
    assert decision.allow is False
    assert decision.projected_tokens == 10
    assert decision.projected_millis == 0
    assert "calibrate" in decision.reason

Put both files in the same directory and run the tests, then a three-round sketch, then the cold-start door on its own. You want the second round refused and the third never considered.

python -m pytest test_round_fence.py -q
python - <<'PY'
from round_fence import Ceiling, Observation, drive
ceiling = Ceiling(max_tokens=100, max_millis=10_000)
rounds = [
    Observation(40, 10, 200, 0),
    Observation(40, 10, 200, 80),
    Observation(40, 10, 200, 80),
]
print(drive(rounds, ceiling))
PY
python - <<'PY'
from round_fence import Ceiling, admit
print(admit(Ceiling(100, 1000), 0, 0, None, 40))
PY

If the files match the listings, pytest should pass, the sketch should print refused_index 1 with a token-ceiling reason and projected_tokens 120, and the cold-start call should refuse with projected_tokens 10 under the default divisor. That printout is the save. The unsent round never left the process, so it could not stall, and it could not call a tool. Compare the projection with a real usage line only after a separate calibration send. If they diverge, fix the divisor or stop projecting. Do not paper over the gap by moving the loop onto a lane whose price reads zero.

Keep the ceiling beside the run, not in a dashboard you will open tomorrow. A small file is enough for a single-operator script. The numbers below are placeholders for your own limit, not a recommendation and not a benchmark.

{
  "max_tokens": 8000,
  "max_millis": 20000,
  "bytes_per_token": 4
}

Load them into Ceiling and log every decision, including the ones you allow. A week later the interesting field is refused_index. If you never refuse, the ceiling is decoration. If you refuse on the second round every time, the task is the wrong shape for a loop. Split the ticket. Hunting a cheaper model will not fix a fan-out you refused to count.

Where a free lane fits

You still need a place to learn what one round costs. That is a measurement job, not a production fan-out. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that measurement, if you already have them: one script, read-only tools, a ceiling written down, and a stop.

This note does not name models, quotas, hardware, or how long that access lasts. Those facts move, and a post that invents them is stale on arrival. Read the current terms before you depend on the lane. If it stalls or vanishes mid-run, the fence in your process is what stops the next call. The lane will not stop it for you.

Use that lane for a single-threaded calibration whose tools only read. Do not use it for a caller who cannot slip a deadline. Do not use it to widen the loop because the dollar field is empty. Do not use it as a second home for a round you already decided was too expensive. Relocating a round is not the same as refusing it. Do not aim a tool that writes, charges, or deletes at a lane you do not control, then act surprised when a slow completion applies the write late. The fence cannot unwind a tool that already committed.

Limits, and who should skip it

The projection is a heuristic. It assumes the next prompt is at least as large as the last one, plus a crude byte split. Real tokenizers disagree with a divide-by-four rule, sometimes by a wide margin. A completion can run longer than the previous one if the model starts a list and will not stop. Elapsed time on a free server is not a sample of any other lane, and one sample is not a distribution.

This fence does not reserve capacity. It does not replace a provider spend cap. It does not see tokens burned inside a tool that calls a model on its own. If a run can spawn a child loop, put the same check in the child, and pass the remaining budget down. Otherwise the parent ceiling is theater: the child spends, and the parent only counts the wrapper.

Skip this pattern when you need invoice-grade reconciliation. Pull the provider usage export and match that. Do not pretend the dataclass is a ledger. Skip it when a late refusal is worse than an over-ceiling round, for example a migration that must finish once it has locked a row. In that case use a hard provider cap and a human checkpoint. Skip it when you have no observation and you are tempted to invent a default, then ship the default as if it were a measurement. The cold-start refusal is the point. Calibrate, then loop.

If you already use MonkeyCode, run the local tests before you touch the network. Then send one calibration script through the free model access on the free server option, write down what a round actually returned, and leave production fan-out behind the fence in your own code. A quiet meter is still not a wider loop.

No Socket Without a Fresh Envelope

Quinn Li — Thu, 08 Oct 2026 08:06:01 +0000

You do not earn a model call by noticing that a free lane looks idle. You earn it when a fresh cost envelope still fits the lane, the deadline, and the meter you are willing to owe. Idle capacity is a rumor. An envelope is a constraint you can fail in a unit test before anyone spends a token.

A coat-check ticket is the right picture. It names the coat, the window, and the door. It does not mean the rack still has a hook when you wander back from the sidewalk. A free lane can be a real place to hang a small probe, and it can also be full, slow, or simply the wrong door for work you already promised a person.

This note is the bouncer in front of that door. You stamp a cost envelope, admit the job to a probe lane or a committed lane, and throw the ticket away when the attempt ends. A retry does not walk in on the old stub.

Reserving a place in line is not the same job as pricing the socket. You can hold a slot and still open an unbounded completion. The queue then looks healthy while a transcript you did not mean to authorize keeps the meter warm. The envelope is the missing half. It is the budget around the prompt, not the prompt, and not the reservation.

What has to be on the ticket

The envelope names an input-token cap, an output-token cap, a wall-clock timeout, an attempt number, a lane, and whether a human deadline is attached. Miss any of those and you are guessing. Guessing is how a five-minute experiment becomes an open tab.

Keep the token numbers local and dull. Count input tokens with a tokenizer you already trust, or with a character heuristic you have checked against your own logs. Label the heuristic as a heuristic. A character count is not a vendor invoice. You are refusing to cross a ceiling, not predicting that the answer will be good.

A probe envelope stays narrow on purpose. Short output cap. One attempt. No inherited retry. No customer deadline. A committed envelope may be wider, but it has to name a meter you can actually pay. It does not get to sneak onto a free lane because that lane answered once last week. Last week is not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are relevant only as places you might run this gate, not as a quota, a hardware shape, a duration, or a promise that the lane stays up. This note does not name models or allotments. Those change, and a workflow that bakes in a number you did not re-read today will lie tomorrow. If you use that free access, keep it on probe-class envelopes. Send committed work to a lane whose failure you can explain to the person waiting.

The free server is a reasonable home for the gate process and for an append-only ledger. It is a poor home for the job that must finish before a launch review. A server you do not pay for can be restarted, contended, or withdrawn.

The ledger still has to survive that, or a restart will mint a second admission for the same attempt. Write the verdict before you open the socket. A crash between those two moments is an admission you may still owe. A crash before the verdict is an attempt that never existed.

Refuse in a test, not in production

The module below is a local proposal you can run on Python 3.9 or newer. It does not call a model. It decides whether a call is allowed to exist. Admission is cheaper than a transcript, and you can run the check on a laptop before you point it at any host.

from dataclasses import dataclass
from enum import Enum

class Lane(Enum):
    PROBE = 'probe'
    COMMITTED = 'committed'

class Decision(Enum):
    ADMIT = 'admit'
    REFUSE = 'refuse'

@dataclass(frozen=True)
class Envelope:
    job_id: str
    attempt: int
    input_token_cap: int
    output_token_cap: int
    timeout_s: int
    lane: Lane
    has_deadline: bool
    meter_named: bool
    issued_at_s: int

@dataclass(frozen=True)
class Verdict:
    decision: Decision
    reason: str

PROBE_OUTPUT_CAP = 512
PROBE_TIMEOUT_S = 20
TICKET_TTL_S = 30

def admit(env: Envelope, now_s: int, seen: set[tuple[str, int]]) -> Verdict:
    key = (env.job_id, env.attempt)
    if key in seen:
        return Verdict(Decision.REFUSE, 'attempt already admitted')
    age = now_s - env.issued_at_s
    if age < 0 or age > TICKET_TTL_S:
        return Verdict(Decision.REFUSE, 'envelope expired or clock skew')
    if env.attempt < 1:
        return Verdict(Decision.REFUSE, 'attempt must start at 1')
    if env.input_token_cap <= 0 or env.output_token_cap <= 0 or env.timeout_s <= 0:
        return Verdict(Decision.REFUSE, 'caps and timeout must be positive')
    if env.lane is Lane.PROBE:
        if env.has_deadline or env.attempt > 1:
            return Verdict(Decision.REFUSE, 'probe lane rejects deadlines and retries')
        wide = env.output_token_cap > PROBE_OUTPUT_CAP or env.timeout_s > PROBE_TIMEOUT_S
        if wide:
            return Verdict(Decision.REFUSE, 'probe envelope too wide')
        return Verdict(Decision.ADMIT, 'probe lane')
    if not env.meter_named:
        return Verdict(Decision.REFUSE, 'committed lane needs a named meter')
    return Verdict(Decision.ADMIT, 'committed lane')

Read it as a bouncer, not as a scheduler. The bouncer does not care that another room is advertising a free hour. The bouncer cares that this ticket is fresh, unused, and matched to this door. The constants are your policy, not a vendor fact. If your logs show probes dying at eight seconds, do not keep twenty because a sample used twenty.

A retry mints a new envelope. Attempt two does not inherit attempt one. If the first attempt died on the probe lane, the second still has to qualify alone. That is how a timeout stops becoming a loop that spends the same prompt again. You can move attempt two onto the committed lane, name a meter, and widen the caps. You cannot stare at an idle free lane and call that a plan.

Three lines, three reasons

Save the module as admit.py. Run a probe that should pass, a retry that should die on the probe lane, and a committed job with no meter. You want three lines. A dashboard can wait until the reasons are boring.

python - <<'PY'
from admit import Envelope, Lane, Decision, admit

now = 1_700_000_000
seen = set()
cases = [
    Envelope('job-7', 1, 800, 256, 15, Lane.PROBE, False, False, now),
    Envelope('job-7', 2, 800, 256, 15, Lane.PROBE, False, False, now),
    Envelope('job-9', 1, 4000, 1200, 60, Lane.COMMITTED, True, False, now),
]
for env in cases:
    verdict = admit(env, now, seen)
    print(env.job_id, env.attempt, verdict.decision.value, verdict.reason)
    if verdict.decision is Decision.ADMIT:
        seen.add((env.job_id, env.attempt))
PY

You should see the probe admitted, the retry refused, and the committed job refused because no meter is named. Rebuild attempt two on the committed lane with the meter flag set, and the gate admits it. That is the move. Change the lane, or change the envelope. Do not widen the free door because the caller is impatient.

When a refusal surprises you, debug the ticket before you debug the model. Print the issue time, the clock you passed in, and the attempt key. An expired envelope looks like a flaky provider if you only read the HTTP timeout.

A reused attempt key looks like a dropped response if you only read the client log. Append one ledger row per verdict: job id, attempt, lane, reason, both caps, and the clock. You can reconstruct a bad night from those rows without opening a completion.

Clock skew belongs in that row. The sample treats a negative age as a refuse, because a ticket from the future is not fresher. It is a lie about time.

If the host running the gate and the host stamping the envelope disagree by a minute, a thirty-second TTL refuses everything and you will blame the model. Stamp issued_at_s on the same host that calls admit, or sync the clocks first. Do not paper over the refuse by raising the TTL until the bug disappears. A long TTL is how a retry inherits a mood from an hour ago.

There is a second failure that looks like thrift. You set a tiny output cap, the gate admits the probe, and the client ignores the cap when it builds the request. The dataclass cannot save you from that.

Enforce the cap in the request builder. If the provider will not take a hard max, truncate the prompt and refuse to send when the counted input exceeds input_token_cap. A gate that only decorates a too-large call is theater.

Queue time is part of the ticket's age, not a free waiting room. If you enqueue first and admit later, the TTL expires in line and you have invented a refusal that feels like an outage. Admit, write the ledger, then enqueue. If the line is longer than the TTL, the job was never eligible for that lane. Lengthen the TTL only when you also lengthen what you are willing to owe if the worker dies mid-call. Time spent waiting is not a discount.

Who should walk away

Do not use this pattern for work that must complete. A probe lane has no deadline by construction. If the caller has a pager, a checkout, or a contract, the free lane is the wrong bet even when it looks empty. Empty is not reserved. A free lane is a place to learn the shape of a call, not a place to hide a launch.

Skip it when you cannot state an output cap. Letting the model decide how long to talk is an open tab, not an envelope. Skip it for inputs you have not chunked, such as a raw repository you hope will fit. The gate will admit a lie if you type a small cap over a huge prompt and then ignore the cap at call time.

The estimator can be wrong in both directions. A tight cap truncates a useful answer. A loose cap can still fit the lane and waste the attempt. This gate does not score quality. It only stops an attempt you did not mean to authorize. If you need evals, give them their own probe-class envelopes. Do not hide an evaluation suite inside a retry loop and call the suite free.

Nothing here measures a provider, names a model, or claims a lasting allotment. If free access shrinks, or the free server goes away, the policy should still refuse a deadline on the probe lane. The numbers in the sample are local. Change them to match the failure you have actually seen, not the failure a blog found convenient.

If you want a free server under the gate and free model access limited to probe-class envelopes, MonkeyCode is one place that currently offers both. Read the live terms before you depend on either. The gate still has to say no when the ticket does not fit.

Don't Enqueue What You Cannot Reserve

Quinn Li — Thu, 24 Sep 2026 21:16:48 +0000

A free lane is the wrong bet when you cannot reserve the attempt before anything admits it. A zero price does not make the prompt free of time, and a client timeout does not hand the tokens back.

You already know the paid version of this failure. The call returns a transcript, the meter moves, and you argue with the bill after the fact. The free version only feels different because no invoice shows up. It is not different in the way that matters. Something still read the prompt, something still held a slot, and your next retry still starts from a smaller remainder than the dashboard in your head.

Think of a counter that comps the drink and still makes you stand in line. The coupon changes who pays. It does not un-grind the beans if you walk out after the order was fired, and it does not give you the meeting you missed while you waited. Free model access works like that coupon. Useful, limited, and easy to misuse if you treat admission as a result.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A free model path, and a free server option beside it, is a reasonable place to practice the habit below. It is not a committed slot, and this note does not pretend otherwise.

What the attempt actually spends

Every attempt spends four things, on a free lane or a paid one. It spends input when the prompt leaves your process. It spends output if generation actually starts. It spends wait between admit and the first usable token. It spends a second attempt if your client treats silence as "that never happened."

The fourth spend is the leak. Your HTTP client gives up at 30 seconds. The server may still be generating, or still sitting in a queue whose clock you do not own. You retry because the spinner looked stuck. Now one user action owns two attempts, and you can merge at most one of them. Without an attempt id, you cannot even say which transcript belongs to which click.

Free capacity makes the leak worse only because it feels safe to ignore. You tell yourself the duplicate was free. The duplicate still occupied the slot your real job needed, and it still burned the context you will not resend cheaply once the window is gone. Queue time was never a refundable deposit. Neither is a prompt that already left the building.

Reserve before you are admitted

The fix is dull, which is why it holds. Before you enqueue, estimate the input, reserve an output cap, and add a slack term for the wait you are willing to abandon. If that reservation does not fit in the job's remaining budget, you do not join the queue. You fail closed in your own process, where failure is still cheap.

The sketch below is a proposal, not a measured run. It uses a character heuristic, not a vendor tokenizer. Swap in the tokenizer you actually ship before you trust a number. The control flow is the artifact: one reservation, one attempt id, no sibling call from a timeout.

import hashlib
import time
from dataclasses import dataclass

@dataclass
class Budget:
    remaining_tokens: int
    max_wait_s: float
    max_attempts: int

@dataclass
class Attempt:
    attempt: str
    reserved: int
    deadline: float
    status: str  # reserved | abandoned | done

def estimate_tokens(text: str) -> int:
    # Stand-in only. Replace with the tokenizer for the model you call.
    return max(1, (len(text) + 3) // 4)

def reserve(prompt: str, output_cap: int, slack_tokens: int) -> int:
    return estimate_tokens(prompt) + output_cap + slack_tokens

def attempt_id(job_id: str, prompt: str, n: int) -> str:
    raw = f"{job_id}|{n}|{prompt}".encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def admit(job_id, prompt, budget, output_cap, slack, n):
    need = reserve(prompt, output_cap, slack)
    if n >= budget.max_attempts or need > budget.remaining_tokens:
        return {"action": "refuse", "need": need,
                "remaining": budget.remaining_tokens}
    budget.remaining_tokens -= need
    att = Attempt(
        attempt=attempt_id(job_id, prompt, n),
        reserved=need,
        deadline=time.time() + budget.max_wait_s,
        status="reserved",
    )
    return {"action": "enqueue", "attempt": att}

def on_timeout(att: Attempt, budget: Budget):
    # Silence is not a new job. Do not call admit() from this branch.
    att.status = "abandoned"
    return {"action": "stop", "attempt": att.attempt,
            "remaining": budget.remaining_tokens}

def on_usage(att: Attempt, budget: Budget, actual: int):
    # Give back unused reserve, or eat the overage if the estimate lied.
    delta = att.reserved - actual
    budget.remaining_tokens += delta
    att.status = "done"
    return {"action": "reconcile", "delta": delta,
            "remaining": budget.remaining_tokens}

You call admit once per user action, not once per socket error. If it returns refuse, you stop. You do not "just try the free server and see." Seeing is the spend. A timeout calls on_timeout and leaves the attempt id alone. A later response, if one still arrives, reconciles against that same id. It does not open a second reservation.

python - <<'PY'
from pathlib import Path
# paste the sketch above, then:
b = Budget(remaining_tokens=1200, max_wait_s=20, max_attempts=1)
prompt = Path("prompt.txt").read_text()
first = admit("job-19", prompt, b, output_cap=400, slack=200, n=0)
print(first["action"])
if first["action"] == "enqueue":
    print(on_timeout(first["attempt"], b)["action"])
second = admit("job-19", prompt, b, output_cap=400, slack=200, n=1)
print(second["action"], second["remaining"])
PY

Read that as a spec, not as a benchmark. The first call may enqueue and subtract the reserve. The timeout marks that attempt abandoned and does not refund the wait you already chose to risk. The second call should refuse, because max_attempts is 1. If your production client does the opposite — retry on timeout, then retry on disconnect, then retry because the UI looked frozen — you have rebuilt the leak in three places, and a free lane will not save you from it.

Slack is the part people set to zero and then regret. Some of the reservation is not tokens you expect to see in a usage field. It is the cost of being wrong about wait. If you cannot name a number you are willing to abandon, you do not have a budget. You have a hope that the queue will be kind.

Where an unknown wait belongs

A free server option is a lane with an unknown wait, not a discount you sprinkle on every call. Use it for work that already has a refuse path. Batch labeling you can defer. A doc summary nobody is staring at. A fixture rebuild that can miss this hour. Put the same reservation in front of that lane that you would put in front of a paid one. The numbers in the budget change. The shape does not.

Do not park a release gate there. A pipeline that must go green before you tag a build cannot wait-and-see on a queue you do not schedule. Do not park an interactive path there either, unless you are willing to show a refusal instead of a spinner. A spinner that hides a second attempt is how one click becomes two spends and one angry refresh.

If the path you need is a guaranteed slot, a named quota, or a latency you can put in an SLO, stop. This workflow will not grow those properties out of a free option. Operator-supplied availability is not a capacity plan. Treat any free model route, including one you might try in MonkeyCode, as optional capacity for jobs that are allowed to miss.

Who should skip the sketch

Skip it when the call is single-shot, tiny, and already capped by a gateway you trust. A 200-token classification that cannot retry does not need a second budget object wrapped around it. Skip it when you cannot estimate input size at all. A reservation built on a lie is just a new surprise with extra steps. Skip it when the job is allowed to block a person. Blocking work needs a committed lane and a cancel the server actually honors. A local deadline field is not a cancel, and abandoning the attempt in your process does not prove the other side stopped reading.

The heuristic will drift. Tool output, images, and a retried prefix blow past len(text)//4. Use the estimate as a gate, then reconcile with on_usage when a usage field comes back. If the response has no usage field, you are flying on the reservation alone, so the cap gets stricter, not looser. Do not "fix" a missing usage field by raising max_attempts. That is the leak again, wearing a config flag.

There is a second failure mode the sketch does not solve. Two workers can admit the same job_id if they do not share the Budget. A process-local counter is enough for a single client and a single job. It is not a distributed lock. If you fan out, the remaining-token number has to live where all of those callers can decrement it, or you will reserve the same tokens twice and call it parallelism.

The habit worth keeping

You do not win this by hunting a cheaper name on a model list. You win it by refusing to enqueue an attempt you have not reserved, and by refusing to mint a new attempt when the old one merely went quiet. Free capacity is a fine place to practice that, because a refusal there costs you a dropped job instead of a surprise invoice. It is a bad place to bet a deadline you cannot miss.

Start with one droppable job. Keep max_attempts at 1. Log the refuse branch next to the enqueue branch, and do not celebrate a spinner that never printed stop. If the refuse count is zero for a week, your cap is decorative, and the queue is still spending prompts you never agreed to reserve.

Don't Put a Retry Loop on Free Capacity

Quinn Li — Wed, 23 Sep 2026 17:08:59 +0000

The first failed call is not your cost problem. The retry is. Free capacity looks like a gift until a tool error, a timeout, or a flaky compile turns one prompt into a chain, and the chain writes a bill you never put in the ticket.

You already treat tokens as a line item. Treat retries as a multiplier on that line item. Every extra attempt resends the conversation, waits again, and can fail for a different reason. That is not a second chance. That is a second invoice stacked on the first.

A coding agent is not a function call. It is a loop with memory. You ask it to patch a test. It calls the compiler. The compiler returns a wall of errors. The model reads the wall, calls the compiler again, then again. You see “it is still working.” The meter sees three full transcripts and two queue delays. If that loop sits on free, interruptible capacity, you are betting a committed job on a spot lane.

The analogy is a taxi that charges for the ride and for every U-turn. The destination did not change. The path did. You still pay for the extra miles.

Finish is a boolean. Cost is not. For any assistant job that can retry, you need three numbers on every attempt: tokens in, tokens out, and seconds spent waiting. Then you need a fourth number after the job ends: the retry multiplier. That is total tokens divided by the tokens of the successful path only. If the successful path is 4,000 tokens and the job burned 18,000, the multiplier is 4.5. You did not buy one answer. You bought four and a half.

Queue time belongs in the same ledger, but not as a separate sermon about waiting. Here the wait is fuel for the next retry. Free lanes stall. A stall is not a refund. While you wait, the ticket is still open, the human is still context-switching, and a later retry may need a warmer, larger prompt because the first error aged out of working memory. The longer the gap, the more the model re-reads, restates, and re-sends.

Retries also poison the transcript. A failed tool result stays in context unless you strip it. The next call is larger. The call after that is larger still. You are not looping on a constant prompt. You are looping on a snowball. That snowball is why “one more try” on a gift lane is an ops decision, not a mood.

Do not guess the multiplier. Log it. The script below is a method, not a production study. It does not call a vendor and it does not claim a fleet result. It records attempts you already made, or rows you simulate, and it prints the multiplier plus a verdict. Wire it to your runner. If you cannot fill the fields, you are not ready to put the job on a shared free lane.

#!/usr/bin/env python3
"""Retry ledger: measure the multiplier, not the last status.

This is an accounting method. Plug real attempt rows from your
agent runner. Do not treat the sample rows as a benchmark.
"""
from __future__ import annotations

import json
from dataclasses import dataclass, asdict
from typing import List, Optional


@dataclass
class Attempt:
    job_id: str
    attempt: int
    tokens_in: int
    tokens_out: int
    queue_s: float
    outcome: str  # ok | tool_error | timeout | cancelled
    keep_error_in_context: bool


def tokens(a: Attempt) -> int:
    return a.tokens_in + a.tokens_out


def multiplier(attempts: List[Attempt]) -> Optional[float]:
    success = [a for a in attempts if a.outcome == "ok"]
    if not success:
        total = sum(tokens(a) for a in attempts)
        return None if total == 0 else float("inf")
    useful = tokens(success[-1])  # last ok path, not the debris before it
    total = sum(tokens(a) for a in attempts)
    return total / useful if useful else float("inf")


def queue_tax(attempts: List[Attempt]) -> float:
    return sum(a.queue_s for a in attempts)


def context_creep(attempts: List[Attempt]) -> int:
    if not attempts:
        return 0
    return tokens(attempts[-1]) - tokens(attempts[0])


def verdict(mult: Optional[float], queued: float, creep: int) -> str:
    if mult is None:
        return "no_success_stop_the_loop"
    if mult == float("inf"):
        return "no_useful_tokens_do_not_retry_on_free"
    if mult >= 3.0 or queued >= 30.0 or creep >= 2000:
        return "free_capacity_is_the_wrong_bet"
    if mult >= 1.8:
        return "cap_retries_and_strip_errors"
    return "free_lane_ok_for_rehearsal_only"


def report(attempts: List[Attempt]) -> dict:
    mult = multiplier(attempts)
    queued = queue_tax(attempts)
    creep = context_creep(attempts)
    return {
        "job_id": attempts[0].job_id if attempts else None,
        "attempts": len(attempts),
        "total_tokens": sum(tokens(a) for a in attempts),
        "retry_multiplier": mult,
        "queue_seconds": queued,
        "context_creep_tokens": creep,
        "verdict": verdict(mult, queued, creep),
        "rows": [asdict(a) for a in attempts],
    }


# Example rows. Replace with your runner's log. Not a claim about any model.
SAMPLE = [
    Attempt("job-77", 1, 1800, 420, 4.0, "tool_error", True),
    Attempt("job-77", 2, 2400, 510, 11.0, "timeout", True),
    Attempt("job-77", 3, 3100, 680, 9.0, "ok", True),
]


if __name__ == "__main__":
    print(json.dumps(report(SAMPLE), indent=2))

Run it as a command, not as a vibe check.

python3 retry_ledger.py
python3 retry_ledger.py | jq '{multiplier: .retry_multiplier, verdict, queue: .queue_seconds, creep: .context_creep_tokens}'

On the sample you should see a multiplier above 2, a queue tax around 24 seconds, and a verdict that free capacity is the wrong bet. Those numbers are a shape. Swap them for your log. If your runner cannot emit tokens_in, tokens_out, and queue_s per attempt, stop. You cannot manage a meter you do not read.

Walk the sample like an incident, not like a demo. Attempt 1 failed with a tool error and still kept the error in context. Attempt 2 therefore started heavier, then timed out after a longer wait. Attempt 3 “succeeded” on a still larger prompt. The last status is green. The ledger is not. That is the whole lesson. Status is what you tell standup. The multiplier is what you owed the meter.

If your agent writes JSONL, fold the same functions over the file. Do not wait for a monthly invoice to discover the loop.

python3 - <<'PY'
import json, retry_ledger as rl
rows = []
with open("attempts.jsonl") as f:
    for line in f:
        d = json.loads(line)
        rows.append(rl.Attempt(**d))
print(json.dumps(rl.report(rows), indent=2))
PY

Before attempt n+1, print the running multiplier. If it crosses 3, stop. Do not “just try once more” on a gift lane. That extra try is how a rehearsal becomes unbudgeted production spend, even when the sticker price is zero, because zero is not the only cost. Human time, lock duration, and a polluted transcript all move.

A tiny guard in the runner looks like this. Keep it boring. Boring is what cost control is supposed to feel like.

MAX_MULT = 3.0
MAX_ATTEMPTS = 2

def should_retry(attempts):
    if len(attempts) >= MAX_ATTEMPTS:
        return False
    m = multiplier(attempts)
    if m is None or m >= MAX_MULT:
        return False
    last = attempts[-1]
    if last.outcome in {"timeout", "cancelled"}:
        return False  # timeouts retry poorly on contended free lanes
    if last.keep_error_in_context and last.tokens_in > 2500:
        return False  # snowball already started
    return last.outcome == "tool_error"

Notice the timeout rule. A tool error sometimes pays for a retry if you strip the huge stderr and keep a one-line cause. A timeout on free capacity usually means the lane is busy or preempted. Retrying a timeout on the same lane is how you donate more queue time to a job that already told you it cannot finish.

Strip before you loop. That is the cheapest retry you will ever buy. Keep the command, the exit code, and one diagnostic line. Drop the rest of the compiler novel. If you cannot strip, you should not retry, because the next prompt is a worse version of the last one. You would not paste three failed CI logs into a new ticket and call that “context.” Do not let the agent do it either.

def compact_tool_error(stderr: str, exit_code: int, cmd: str) -> str:
    lines = [ln.strip() for ln in stderr.splitlines() if ln.strip()]
    last = lines[-1] if lines else "(no stderr)"
    return f"{cmd} exited {exit_code}: {last[:240]}"

Use free capacity when the job is a rehearsal: a prompt you are still shaping, a script you are still wiring, a one-off question you can drop. Kill it when the multiplier climbs. Do not promote that lane to “the way we ship the patch.”

Refuse the free lane when the work has a human waiting on it. Refuse it when the loop can mutate a repo or an environment. Refuse it when the prompt is already large, when the last failure was a timeout or a cancel, or when you cannot log tokens per attempt. In those cases the cheap lane is the expensive decision. The retry multiplier turns a gift into a stall, and the stall still burns context.

Think of free capacity as a bus that stops when it is full. Committed work is a seat you bought. You do not put a deploy-blocking test fix on the bus and then act surprised when it gets bounced and you buy three more tickets trying to arrive on time. Spot is for experiments you can abandon. A retry loop is a promise that you will not abandon. Those two facts do not share a lane.

If you need a box to practice the ledger itself, MonkeyCode’s free model access and free server option are enough to run the accounting script and a short, capped rehearsal. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That is the right size of bet: learn the multiplier on a scratch box. It is the wrong size of bet for an unattended retry loop against a live branch.

This ledger does not know a vendor’s billing granularity. Some meters round. Some count cached prefixes differently. Some charge for tool payloads you never see in the chat UI. If you cannot map a row in the log to a row on the invoice, treat the multiplier as a lower bound. The sample rows are not a benchmark. Do not cite 4.5x, 24 seconds, or a threshold of 3.0 as measured results from a fleet. Set the cuts from your own traces. A research spike and a CI fixer are not the same job.

The verdict function is conservative on purpose. It will tell you to leave free capacity earlier than a hopeful operator would. That is the point. Hope is not a cost control. A green last attempt can still be a bad buy if the path to green was three transcripts and a queue.

Who should not use this approach: anyone who cannot instrument attempts; anyone running unattended agents against production credentials; anyone who needs a hard SLA and thinks a free lane will provide one; anyone shopping for a model beauty contest. This is ops arithmetic. It will not pick a model for you. It will tell you when to stop looping.

If the job matters, cap the retries, strip the errors, and put the loop on capacity you actually reserved. Free is for learning the shape of the bill. It is not for surviving the loop.

Don't Rehearse a Bill on Gift Capacity

Quinn Li — Tue, 22 Sep 2026 15:23:06 +0000

You cannot price a paid path by running the same prompt on gift capacity. A free queue has a different wait, a different preemption story, and a different retry surface. Treat it as a scratch pad. Rehearse spend only where the invoice will actually land.

That sounds obvious until a tool-calling agent walks in. One user message looks cheap. Then the model asks for a tool, you stuff the JSON back into the transcript, the context grows, and the "same" job is four prompts with a fatter tail.

On gift capacity you shrug. In production that tail is the bill.

Think of it like timing a delivery van in a bicycle lane. You will get a number. It will be a number about bicycles.

The usual failure is not a mysterious model. It is a staging habit. You point the same prompt text at a free server because the string matches, then you treat latency, retries, and token mix as if they transferred. Matching text is not matching economics. Cold start, boarding delay, mid-loop preemption, and the ratio of input tokens to output tokens all move when the iron and the meter diverge. Your canary starts lying the moment those two diverge.

Label the work before you label the model.

Every job needs a lane. exploratory may ride gift capacity while you are still wrong on purpose. rehearsal must hit the endpoint, the token cap, the tool budget, and the concurrency you will ship. committed is user-facing or tied to a ship clock. If you cannot name the lane, you are already in the wrong one. A prompt without a lane is how a sketch inherits a production retry ceiling, and how a production agent inherits a gift queue.

Here is a small gate you can run locally. It is a classifier, not a FinOps platform. It refuses to send rehearsal or committed work at a gift base URL. It also refuses to pretend a character heuristic is a vendor quote. Read the comments in the file before you trust a single integer it prints.

#!/usr/bin/env python3
"""job_lane.py — refuse to rehearse a bill on gift capacity.

Heuristic only. Ceiling, not a quote. Not a vendor invoice.
"""
from __future__ import annotations

import argparse
import json
import os
import sys
from dataclasses import dataclass
from typing import Literal

Lane = Literal["exploratory", "rehearsal", "committed"]

@dataclass
class JobEnvelope:
    job_id: str
    lane: Lane
    prompt: str
    tool_hops: int = 0
    max_tokens: int = 512
    concurrency: int = 1
    retry_ceiling: int = 1
    base_url: str = ""

    def heuristic_input_tokens(self) -> int:
        # char/4 stand-in. Not a tokenizer. Do not invoice from this.
        return max(1, len(self.prompt) // 4)

    def output_ceiling(self) -> int:
        hops = max(1, self.tool_hops + 1)
        return self.max_tokens * hops

def is_gift_endpoint(base_url: str) -> bool:
    marker = os.environ.get("GIFT_BASE_URL", "").rstrip("/")
    url = (base_url or os.environ.get("LLM_BASE_URL", "")).rstrip("/")
    if not url:
        return False
    if marker and url == marker:
        return True
    lowered = url.lower()
    return "gift" in lowered or "free" in lowered

def gate(job: JobEnvelope) -> dict:
    gift = is_gift_endpoint(job.base_url)
    input_tok = job.heuristic_input_tokens()
    out_ceil = job.output_ceiling()
    ceiling = (input_tok + out_ceil) * max(1, job.retry_ceiling)
    allowed, reason = True, "ok"

    if job.lane in ("rehearsal", "committed") and gift:
        allowed, reason = False, "refusing to rehearse or ship on gift capacity"
    elif job.lane == "committed" and job.retry_ceiling > 2:
        allowed, reason = False, "committed work cannot hide behind a wide retry ceiling"
    elif gift and job.concurrency > 1:
        allowed, reason = False, "concurrency on gift capacity is not a load test"

    return {
        "job_id": job.job_id,
        "lane": job.lane,
        "gift_endpoint": gift,
        "allowed": allowed,
        "reason": reason,
        "heuristic_input_tokens": input_tok,
        "heuristic_output_ceiling": out_ceil,
        "retry_ceiling": job.retry_ceiling,
        "heuristic_token_ceiling": ceiling,
        "note": "ceiling is a tripwire, not a vendor quote",
    }

def simulate_tool_loop(user_msg: str, tool_payloads: list[str]) -> dict:
    """The first message is not the meter. Each tool result is more prompt."""
    transcript = user_msg
    hops = []
    for i, payload in enumerate(tool_payloads, start=1):
        transcript += "\n" + payload
        hops.append({
            "hop": i,
            "transcript_chars": len(transcript),
            "heuristic_tokens": max(1, len(transcript) // 4),
        })
    return {
        "first_message_heuristic_tokens": max(1, len(user_msg) // 4),
        "final_transcript_heuristic_tokens": max(1, len(transcript) // 4),
        "hops": hops,
    }

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--job", required=True)
    parser.add_argument("--simulate-tools", action="store_true")
    args = parser.parse_args()
    raw = json.loads(open(args.job, encoding="utf-8").read())
    job = JobEnvelope(**{k: raw[k] for k in JobEnvelope.__dataclass_fields__ if k in raw})
    result = gate(job)
    if args.simulate_tools:
        tools = raw.get("tool_payloads", [])
        result["tool_loop"] = simulate_tool_loop(job.prompt, tools)
    print(json.dumps(result, indent=2))
    if not result["allowed"]:
        print(result["reason"], file=sys.stderr)
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Save a fixture that pretends to be a rehearsal. Notice the lane and the base URL disagree on purpose. That disagreement is the bug you want CI to catch, not a clever default you want the model to paper over.

{
  "job_id": "agent-rehearsal-014",
  "lane": "rehearsal",
  "prompt": "List open invoices, then draft a dunning note.",
  "tool_hops": 3,
  "max_tokens": 800,
  "concurrency": 4,
  "retry_ceiling": 3,
  "base_url": "https://gift.example.invalid/v1",
  "tool_payloads": [
    "{\"tool\":\"sql\",\"rows\": 42, \"csv\": \"...\"}",
    "{\"tool\":\"crm\",\"notes\": \"prior balance, three emails\"}",
    "{\"tool\":\"policy\",\"tone\": \"firm\"}"
  ]
}

Run the gate twice. First as a reject. Then point base_url at the path you actually pay, drop concurrency to what you will ship, and run it again as a ceiling check. The second run is still not an invoice. It is a tripwire you can fail a build with.

python3 job_lane.py --job rehearsal.json --simulate-tools; echo exit:$?

# expected: exit 2, reason about gift capacity

GIFT_BASE_URL="https://gift.example.invalid/v1" \
LLM_BASE_URL="https://gift.example.invalid/v1" \
python3 job_lane.py --job rehearsal.json; echo exit:$?

The --simulate-tools path is the part most cost dashboards still flatten. Each tool result is not a side channel. It is more prompt. If you only count the first user message, you are counting the first mile of a round trip and calling it the route. Watch the hop list grow. The last hop is where output tokens get expensive, because the model is now reading its own homework plus yours.

That is why gift capacity is the wrong rehearsal. Preemption in the middle of a four-hop tool loop does not just waste time. It often restarts the graph. You already paid in calendar, and on a metered path you may pay in tokens for the partial transcript too. The free path hides that, because you are not looking at an invoice. You are looking at a courtesy.

Queue time still burns calendar. Calendar is what a release train spends. A gift server that boards you in twenty minutes is not a rehearsal of a 400ms p95. It is a different sport with the same jersey. Load tests on that sport will teach finance the wrong lesson, and they will teach you to overfit a queue you will not have on Monday morning.

So split the lanes on purpose, in the job envelope, not in a wiki nobody reads at 2 a.m.

Use gift capacity to be wrong cheaply. Draft the prompt. Throw away the transcript. Confirm the tool schema even parses. Confirm the agent does not need seven hops to ask for the date. Then copy the envelope onto the paid path and run a short rehearsal at production concurrency, with the same max_tokens, the same stop conditions, and the same retry ceiling. Record input tokens, output tokens, tool hops, queue wait, and wall time as separate columns. If you fold them into one duration number, you will optimize the wrong column. Duration can look fine while output tokens quietly eat the month.

For the exploratory lane only, MonkeyCode — an open-source project with free model access and a free server option — can hold draft-and-discard work so you do not burn the paid meter while the schema is still wrong. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Park sketches there, then move rehearsal onto the endpoint you will actually invoice.

A limitation worth saying out loud: this gate does not know your vendor's billing rules. Some meters count killed streams. Some count cached prefixes differently. Some round up. Some bill tool payloads at a different rate than the surrounding prose. The char/4 stand-in will drift on code, on CJK, and on compressed JSON. Use it as a tripwire that fails closed when a lane and an endpoint disagree. Do not use it as a quote you paste into a budget slide.

Who should not put work on gift capacity at all: anything carrying secrets or regulated user data; anything you plan to treat as an availability replica; any load test whose numbers will be shown to finance as a forecast. Gift capacity is not a cluster you own. It is not a capacity reserve. It is a courtesy with a different SLO, which is to say, maybe none. If your agent is the product, the free path is the wrong bet for canaries. If your job is still a sketch, burning the paid meter to find a better system prompt is the other wrong bet.

Keep those two mistakes from sharing a queue. Name the lane. Gate the URL. Count every hop. Then believe the invoice, not the gift.

Killed Work Still Hits the Meter

Quinn Li — Mon, 21 Sep 2026 13:39:28 +0000

Free capacity is a discount on the next token, not a refund on the last one. If the box dies, the queue expires, or the call is cut off, you still spent the prompt. You still spent the partial answer. You still spent the minutes a human waited. The kill switch does not roll the meter back.

That is the part most teams skip when they park production-shaped work on a scavenger lane. They see a zero in the rate column and stop reading. The invoice is not the rate. The invoice is every token that never became a finished, accepted result.

Think of a taxi the dispatcher can recall mid-ride. The fare looks cheap until the car vanishes at the third stoplight. You do not get those kilometers back. You stand on the curb with a half-finished trip and you pay again to start over. Model jobs behave the same way. Input tokens are prepaid. Partial output is often billed. A restart repeats the prefix you already paid for.

So you need a different unit. Stop asking whether the lane is free. Ask how many times this job can die before the discount costs more than paying now. That number is an abort budget. Without it, spare capacity is not a strategy. It is a leak with a friendly name.

Two lanes, one promotion rule

You already know the shape of the work. Drafts, eval sweeps, prompt archaeology, and throwaway summaries can die. The nightly batch that writes a customer-facing changelog cannot. The classifier that gates a merge cannot. The job that holds a lock in your deploy graph cannot.

Put killable work on scavenged capacity. Put must-finish work on capacity you actually control. The rule is boring, which is why it survives contact with a real queue. A scavenger lane is still useful. You want somewhere to burn exploratory tokens without contaminating the committed path.

MonkeyCode's free model access and free server option can sit in that lane: a box for work you are willing to lose. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That is not a reserved machine, and it is not a claim about quota, hardware, model names, or how long a server stays yours. Treat it as spare capacity you might not get back. If the job cannot tolerate a restart, promote it before the first token. If you want a place to park killable drafts off the paying queue, that free lane is there to try. Measure aborts before you trust it.

An abort budget you can run

The following is a labeled example, not a production scheduler and not a benchmark. You supply the paid token rate and the cost of your own time. The function answers one question: after this many kills, is waiting on free capacity still cheaper than paying?

# abort_budget.py
# Proposal: promote a job off scavenged capacity when restart
# waste meets or exceeds the paid alternative.
from dataclasses import dataclass

@dataclass
class Job:
    name: str
    must_finish: bool
    max_aborts: int
    input_tokens: int
    expected_output_tokens: int
    queue_s_expected: float
    engineer_usd_per_hour: float
    paid_usd_per_million: float

def tokens_burned_on_kill(job: Job, partial_output: int) -> int:
    # Prompt is spent. Partial completion is spent.
    # A restart pays the prompt again.
    return job.input_tokens + max(partial_output, 0)

def waste_usd(job: Job, kills: int, avg_partial_out: int) -> float:
    token_waste = kills * tokens_burned_on_kill(job, avg_partial_out)
    token_usd = token_waste * job.paid_usd_per_million / 1_000_000
    hours = kills * job.queue_s_expected / 3600.0
    time_usd = hours * job.engineer_usd_per_hour
    return token_usd + time_usd

def paid_path_usd(job: Job) -> float:
    total = job.input_tokens + job.expected_output_tokens
    return total * job.paid_usd_per_million / 1_000_000

def should_promote(job: Job, kills: int, avg_partial_out: int) -> bool:
    if job.must_finish:
        return True
    if kills >= job.max_aborts:
        return True
    return waste_usd(job, kills, avg_partial_out) >= paid_path_usd(job)

The comparison uses your paid rate as the opportunity cost of tokens you already threw away. That is deliberate. A free prompt you later repeat on a paid model was never free. It was a loan. You repay it with a second prefix and whatever waiting you did the first time.

Wire a test so the rule cannot drift into folklore.

# test_abort_budget.py
from abort_budget import Job, should_promote, waste_usd, paid_path_usd

def sample_job(**kwargs) -> Job:
    base = dict(
        name="draft-summary",
        must_finish=False,
        max_aborts=2,
        input_tokens=8_000,
        expected_output_tokens=1_200,
        queue_s_expected=180.0,
        engineer_usd_per_hour=75.0,
        paid_usd_per_million=3.0,
    )
    base.update(kwargs)
    return Job(**base)

def test_must_finish_promotes_immediately():
    job = sample_job(must_finish=True, name="release-notes")
    assert should_promote(job, kills=0, avg_partial_out=0) is True

def test_killable_job_stays_until_budget_breaks():
    job = sample_job()
    assert should_promote(job, kills=0, avg_partial_out=400) is False
    assert should_promote(job, kills=2, avg_partial_out=400) is True

def test_long_queue_promotes_before_token_waste_does():
    job = sample_job(queue_s_expected=2400.0, max_aborts=9)
    # Forty minutes of blocked waiting, twice, is not a discount.
    assert waste_usd(job, 2, 0) > paid_path_usd(job)
    assert should_promote(job, kills=2, avg_partial_out=0) is True

Those dollar figures are illustrations you must replace. They are not product prices, not market rates, and not a claim about your bill. If you do not know the paid rate, stop and export one day of usage from the provider you actually use. A budget with a made-up rate is a story.

python -m pip install pytest
pytest -q test_abort_budget.py

You can also poke the function from a shell without standing up a framework. Keep the numbers in environment variables so a teammate can disagree with the rate instead of editing the rule.

python - <<'PY'
from abort_budget import Job, should_promote, waste_usd, paid_path_usd
job = Job(
    name="eval-sweep",
    must_finish=False,
    max_aborts=3,
    input_tokens=12_000,
    expected_output_tokens=800,
    queue_s_expected=90.0,
    engineer_usd_per_hour=75.0,
    paid_usd_per_million=3.0,
)
print("paid_path", round(paid_path_usd(job), 6))
print("waste_after_2", round(waste_usd(job, 2, 300), 6))
print("promote", should_promote(job, kills=2, avg_partial_out=300))
PY

Read the incomplete calls

The budget only works if you can see kills. Most traces hide them under a generic timeout. You want one line per attempt: job id, attempt number, tokens in, tokens out, terminal state. Terminal state is the whole point. complete, timeout, preempted, and cancelled are different invoices.

# attempt_log.py
# Proposal: parse JSONL attempts and split spent tokens into
# finished work versus killed work.
import json
import sys

def summarize(path: str) -> None:
    spent = 0
    finished = 0
    killed = 0
    with open(path, encoding="utf-8") as handle:
        for raw in handle:
            row = json.loads(raw)
            used = int(row["tokens_in"]) + int(row["tokens_out"])
            spent += used
            if row["state"] == "complete":
                finished += used
            else:
                killed += used
    print(f"spent_tokens={spent}")
    print(f"finished_tokens={finished}")
    print(f"killed_tokens={killed}")
    if spent:
        print(f"killed_ratio={killed / spent:.3f}")

if __name__ == "__main__":
    summarize(sys.argv[1])

Start with a fixture so the parser has something honest to chew on.

cat > attempts.jsonl <<'EOF'
{"id":"draft-summary","attempt":1,"tokens_in":8000,"tokens_out":410,"state":"preempted"}
{"id":"draft-summary","attempt":2,"tokens_in":8000,"tokens_out":0,"state":"timeout"}
{"id":"draft-summary","attempt":3,"tokens_in":8000,"tokens_out":1180,"state":"complete"}
EOF
python attempt_log.py attempts.jsonl

If killed_ratio climbs while must_finish jobs are still on the scavenger lane, you are not saving. You are rehearsing the same prompt. That is cousin to a retry storm, except the trigger is the machine going away rather than the model returning an error. The fix is promotion, not a longer timeout. A timeout is how you donate a larger partial completion to the killed column.

Watch the shape of the waste. A huge prompt with a tiny completion is a bad scavenger candidate, because every kill repeats the expensive half. A short prompt with a long, optional draft is a better one. The log will tell you which job you actually have. Your intuition about "cheap experiments" will not.

When free capacity is the wrong bet

Use the scavenger lane when the result is optional, the prompt is cheap to replay, and nobody is blocked on the output. Eval harnesses, prompt diffs, and personal drafts fit. A free server is a fine place to learn whether a prompt is even worth paying for.

Keep committed work off that lane. If the output enters a customer path, a compliance record, or a deploy gate, the job already has an SLO whether you wrote it down or not. If a human is sitting in a meeting waiting on the result, their time will beat a token discount in a single missed slot. If the prompt is large and the completion is small, a kill repeats the costly half and the discount evaporates on the first preempt.

Do not use spare capacity as capacity planning. A box that can vanish is not a headcount substitute and not an SLO. If you need the job tomorrow morning, you already know the answer. Pay, or cut the scope until the job can die without consequences.

The approach also fails if you cannot attribute tokens per attempt. Shared keys, one giant chat session, and dashboards that hide usage behind a monthly total will lie to you. Fix the log first. Then set max_aborts. Then promote.

Limitations are blunt. This model ignores cache hits, ignores provider-side retries you cannot see, and ignores the chance that the paid path is congested too. It assumes you can move a job between lanes without rewriting it. If your orchestration cannot redirect a single task, the function is a comment, not a control. It also assumes the scavenger lane still costs you nothing per token, which may not remain true, and which you should re-read in the product's own terms rather than in a post.

You should not adopt this if you need a guarantee. You should not adopt it if you cannot kill the job. You should not adopt it if the work is someone else's data you are not allowed to send to a shared box. You should not adopt it if the only number you track is requests, because requests are how incomplete work hides.

The core move is small. Tag the job. Cap the kills. Promote when the waste crosses the paid path. Everything else is decoration on a meter that already started.