DEV Community: Quinn Li

Don't Enqueue What You Cannot Reserve

Quinn Li — Thu, 24 Sep 2026 21:16:48 +0000

A free lane is the wrong bet when you cannot reserve the attempt before anything admits it. A zero price does not make the prompt free of time, and a client timeout does not hand the tokens back.

You already know the paid version of this failure. The call returns a transcript, the meter moves, and you argue with the bill after the fact. The free version only feels different because no invoice shows up. It is not different in the way that matters. Something still read the prompt, something still held a slot, and your next retry still starts from a smaller remainder than the dashboard in your head.

Think of a counter that comps the drink and still makes you stand in line. The coupon changes who pays. It does not un-grind the beans if you walk out after the order was fired, and it does not give you the meeting you missed while you waited. Free model access works like that coupon. Useful, limited, and easy to misuse if you treat admission as a result.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A free model path, and a free server option beside it, is a reasonable place to practice the habit below. It is not a committed slot, and this note does not pretend otherwise.

What the attempt actually spends

Every attempt spends four things, on a free lane or a paid one. It spends input when the prompt leaves your process. It spends output if generation actually starts. It spends wait between admit and the first usable token. It spends a second attempt if your client treats silence as "that never happened."

The fourth spend is the leak. Your HTTP client gives up at 30 seconds. The server may still be generating, or still sitting in a queue whose clock you do not own. You retry because the spinner looked stuck. Now one user action owns two attempts, and you can merge at most one of them. Without an attempt id, you cannot even say which transcript belongs to which click.

Free capacity makes the leak worse only because it feels safe to ignore. You tell yourself the duplicate was free. The duplicate still occupied the slot your real job needed, and it still burned the context you will not resend cheaply once the window is gone. Queue time was never a refundable deposit. Neither is a prompt that already left the building.

Reserve before you are admitted

The fix is dull, which is why it holds. Before you enqueue, estimate the input, reserve an output cap, and add a slack term for the wait you are willing to abandon. If that reservation does not fit in the job's remaining budget, you do not join the queue. You fail closed in your own process, where failure is still cheap.

The sketch below is a proposal, not a measured run. It uses a character heuristic, not a vendor tokenizer. Swap in the tokenizer you actually ship before you trust a number. The control flow is the artifact: one reservation, one attempt id, no sibling call from a timeout.

import hashlib
import time
from dataclasses import dataclass

@dataclass
class Budget:
    remaining_tokens: int
    max_wait_s: float
    max_attempts: int

@dataclass
class Attempt:
    attempt: str
    reserved: int
    deadline: float
    status: str  # reserved | abandoned | done

def estimate_tokens(text: str) -> int:
    # Stand-in only. Replace with the tokenizer for the model you call.
    return max(1, (len(text) + 3) // 4)

def reserve(prompt: str, output_cap: int, slack_tokens: int) -> int:
    return estimate_tokens(prompt) + output_cap + slack_tokens

def attempt_id(job_id: str, prompt: str, n: int) -> str:
    raw = f"{job_id}|{n}|{prompt}".encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def admit(job_id, prompt, budget, output_cap, slack, n):
    need = reserve(prompt, output_cap, slack)
    if n >= budget.max_attempts or need > budget.remaining_tokens:
        return {"action": "refuse", "need": need,
                "remaining": budget.remaining_tokens}
    budget.remaining_tokens -= need
    att = Attempt(
        attempt=attempt_id(job_id, prompt, n),
        reserved=need,
        deadline=time.time() + budget.max_wait_s,
        status="reserved",
    )
    return {"action": "enqueue", "attempt": att}

def on_timeout(att: Attempt, budget: Budget):
    # Silence is not a new job. Do not call admit() from this branch.
    att.status = "abandoned"
    return {"action": "stop", "attempt": att.attempt,
            "remaining": budget.remaining_tokens}

def on_usage(att: Attempt, budget: Budget, actual: int):
    # Give back unused reserve, or eat the overage if the estimate lied.
    delta = att.reserved - actual
    budget.remaining_tokens += delta
    att.status = "done"
    return {"action": "reconcile", "delta": delta,
            "remaining": budget.remaining_tokens}

You call admit once per user action, not once per socket error. If it returns refuse, you stop. You do not "just try the free server and see." Seeing is the spend. A timeout calls on_timeout and leaves the attempt id alone. A later response, if one still arrives, reconciles against that same id. It does not open a second reservation.

python - <<'PY'
from pathlib import Path
# paste the sketch above, then:
b = Budget(remaining_tokens=1200, max_wait_s=20, max_attempts=1)
prompt = Path("prompt.txt").read_text()
first = admit("job-19", prompt, b, output_cap=400, slack=200, n=0)
print(first["action"])
if first["action"] == "enqueue":
    print(on_timeout(first["attempt"], b)["action"])
second = admit("job-19", prompt, b, output_cap=400, slack=200, n=1)
print(second["action"], second["remaining"])
PY

Read that as a spec, not as a benchmark. The first call may enqueue and subtract the reserve. The timeout marks that attempt abandoned and does not refund the wait you already chose to risk. The second call should refuse, because max_attempts is 1. If your production client does the opposite — retry on timeout, then retry on disconnect, then retry because the UI looked frozen — you have rebuilt the leak in three places, and a free lane will not save you from it.

Slack is the part people set to zero and then regret. Some of the reservation is not tokens you expect to see in a usage field. It is the cost of being wrong about wait. If you cannot name a number you are willing to abandon, you do not have a budget. You have a hope that the queue will be kind.

Where an unknown wait belongs

A free server option is a lane with an unknown wait, not a discount you sprinkle on every call. Use it for work that already has a refuse path. Batch labeling you can defer. A doc summary nobody is staring at. A fixture rebuild that can miss this hour. Put the same reservation in front of that lane that you would put in front of a paid one. The numbers in the budget change. The shape does not.

Do not park a release gate there. A pipeline that must go green before you tag a build cannot wait-and-see on a queue you do not schedule. Do not park an interactive path there either, unless you are willing to show a refusal instead of a spinner. A spinner that hides a second attempt is how one click becomes two spends and one angry refresh.

If the path you need is a guaranteed slot, a named quota, or a latency you can put in an SLO, stop. This workflow will not grow those properties out of a free option. Operator-supplied availability is not a capacity plan. Treat any free model route, including one you might try in MonkeyCode, as optional capacity for jobs that are allowed to miss.

Who should skip the sketch

Skip it when the call is single-shot, tiny, and already capped by a gateway you trust. A 200-token classification that cannot retry does not need a second budget object wrapped around it. Skip it when you cannot estimate input size at all. A reservation built on a lie is just a new surprise with extra steps. Skip it when the job is allowed to block a person. Blocking work needs a committed lane and a cancel the server actually honors. A local deadline field is not a cancel, and abandoning the attempt in your process does not prove the other side stopped reading.

The heuristic will drift. Tool output, images, and a retried prefix blow past len(text)//4. Use the estimate as a gate, then reconcile with on_usage when a usage field comes back. If the response has no usage field, you are flying on the reservation alone, so the cap gets stricter, not looser. Do not "fix" a missing usage field by raising max_attempts. That is the leak again, wearing a config flag.

There is a second failure mode the sketch does not solve. Two workers can admit the same job_id if they do not share the Budget. A process-local counter is enough for a single client and a single job. It is not a distributed lock. If you fan out, the remaining-token number has to live where all of those callers can decrement it, or you will reserve the same tokens twice and call it parallelism.

The habit worth keeping

You do not win this by hunting a cheaper name on a model list. You win it by refusing to enqueue an attempt you have not reserved, and by refusing to mint a new attempt when the old one merely went quiet. Free capacity is a fine place to practice that, because a refusal there costs you a dropped job instead of a surprise invoice. It is a bad place to bet a deadline you cannot miss.

Start with one droppable job. Keep max_attempts at 1. Log the refuse branch next to the enqueue branch, and do not celebrate a spinner that never printed stop. If the refuse count is zero for a week, your cap is decorative, and the queue is still spending prompts you never agreed to reserve.

Don't Put a Retry Loop on Free Capacity

Quinn Li — Wed, 23 Sep 2026 17:08:59 +0000

The first failed call is not your cost problem. The retry is. Free capacity looks like a gift until a tool error, a timeout, or a flaky compile turns one prompt into a chain, and the chain writes a bill you never put in the ticket.

You already treat tokens as a line item. Treat retries as a multiplier on that line item. Every extra attempt resends the conversation, waits again, and can fail for a different reason. That is not a second chance. That is a second invoice stacked on the first.

A coding agent is not a function call. It is a loop with memory. You ask it to patch a test. It calls the compiler. The compiler returns a wall of errors. The model reads the wall, calls the compiler again, then again. You see “it is still working.” The meter sees three full transcripts and two queue delays. If that loop sits on free, interruptible capacity, you are betting a committed job on a spot lane.

The analogy is a taxi that charges for the ride and for every U-turn. The destination did not change. The path did. You still pay for the extra miles.

Finish is a boolean. Cost is not. For any assistant job that can retry, you need three numbers on every attempt: tokens in, tokens out, and seconds spent waiting. Then you need a fourth number after the job ends: the retry multiplier. That is total tokens divided by the tokens of the successful path only. If the successful path is 4,000 tokens and the job burned 18,000, the multiplier is 4.5. You did not buy one answer. You bought four and a half.

Queue time belongs in the same ledger, but not as a separate sermon about waiting. Here the wait is fuel for the next retry. Free lanes stall. A stall is not a refund. While you wait, the ticket is still open, the human is still context-switching, and a later retry may need a warmer, larger prompt because the first error aged out of working memory. The longer the gap, the more the model re-reads, restates, and re-sends.

Retries also poison the transcript. A failed tool result stays in context unless you strip it. The next call is larger. The call after that is larger still. You are not looping on a constant prompt. You are looping on a snowball. That snowball is why “one more try” on a gift lane is an ops decision, not a mood.

Do not guess the multiplier. Log it. The script below is a method, not a production study. It does not call a vendor and it does not claim a fleet result. It records attempts you already made, or rows you simulate, and it prints the multiplier plus a verdict. Wire it to your runner. If you cannot fill the fields, you are not ready to put the job on a shared free lane.

#!/usr/bin/env python3
"""Retry ledger: measure the multiplier, not the last status.

This is an accounting method. Plug real attempt rows from your
agent runner. Do not treat the sample rows as a benchmark.
"""
from __future__ import annotations

import json
from dataclasses import dataclass, asdict
from typing import List, Optional


@dataclass
class Attempt:
    job_id: str
    attempt: int
    tokens_in: int
    tokens_out: int
    queue_s: float
    outcome: str  # ok | tool_error | timeout | cancelled
    keep_error_in_context: bool


def tokens(a: Attempt) -> int:
    return a.tokens_in + a.tokens_out


def multiplier(attempts: List[Attempt]) -> Optional[float]:
    success = [a for a in attempts if a.outcome == "ok"]
    if not success:
        total = sum(tokens(a) for a in attempts)
        return None if total == 0 else float("inf")
    useful = tokens(success[-1])  # last ok path, not the debris before it
    total = sum(tokens(a) for a in attempts)
    return total / useful if useful else float("inf")


def queue_tax(attempts: List[Attempt]) -> float:
    return sum(a.queue_s for a in attempts)


def context_creep(attempts: List[Attempt]) -> int:
    if not attempts:
        return 0
    return tokens(attempts[-1]) - tokens(attempts[0])


def verdict(mult: Optional[float], queued: float, creep: int) -> str:
    if mult is None:
        return "no_success_stop_the_loop"
    if mult == float("inf"):
        return "no_useful_tokens_do_not_retry_on_free"
    if mult >= 3.0 or queued >= 30.0 or creep >= 2000:
        return "free_capacity_is_the_wrong_bet"
    if mult >= 1.8:
        return "cap_retries_and_strip_errors"
    return "free_lane_ok_for_rehearsal_only"


def report(attempts: List[Attempt]) -> dict:
    mult = multiplier(attempts)
    queued = queue_tax(attempts)
    creep = context_creep(attempts)
    return {
        "job_id": attempts[0].job_id if attempts else None,
        "attempts": len(attempts),
        "total_tokens": sum(tokens(a) for a in attempts),
        "retry_multiplier": mult,
        "queue_seconds": queued,
        "context_creep_tokens": creep,
        "verdict": verdict(mult, queued, creep),
        "rows": [asdict(a) for a in attempts],
    }


# Example rows. Replace with your runner's log. Not a claim about any model.
SAMPLE = [
    Attempt("job-77", 1, 1800, 420, 4.0, "tool_error", True),
    Attempt("job-77", 2, 2400, 510, 11.0, "timeout", True),
    Attempt("job-77", 3, 3100, 680, 9.0, "ok", True),
]


if __name__ == "__main__":
    print(json.dumps(report(SAMPLE), indent=2))

Run it as a command, not as a vibe check.

python3 retry_ledger.py
python3 retry_ledger.py | jq '{multiplier: .retry_multiplier, verdict, queue: .queue_seconds, creep: .context_creep_tokens}'

On the sample you should see a multiplier above 2, a queue tax around 24 seconds, and a verdict that free capacity is the wrong bet. Those numbers are a shape. Swap them for your log. If your runner cannot emit tokens_in, tokens_out, and queue_s per attempt, stop. You cannot manage a meter you do not read.

Walk the sample like an incident, not like a demo. Attempt 1 failed with a tool error and still kept the error in context. Attempt 2 therefore started heavier, then timed out after a longer wait. Attempt 3 “succeeded” on a still larger prompt. The last status is green. The ledger is not. That is the whole lesson. Status is what you tell standup. The multiplier is what you owed the meter.

If your agent writes JSONL, fold the same functions over the file. Do not wait for a monthly invoice to discover the loop.

python3 - <<'PY'
import json, retry_ledger as rl
rows = []
with open("attempts.jsonl") as f:
    for line in f:
        d = json.loads(line)
        rows.append(rl.Attempt(**d))
print(json.dumps(rl.report(rows), indent=2))
PY

Before attempt n+1, print the running multiplier. If it crosses 3, stop. Do not “just try once more” on a gift lane. That extra try is how a rehearsal becomes unbudgeted production spend, even when the sticker price is zero, because zero is not the only cost. Human time, lock duration, and a polluted transcript all move.

A tiny guard in the runner looks like this. Keep it boring. Boring is what cost control is supposed to feel like.

MAX_MULT = 3.0
MAX_ATTEMPTS = 2

def should_retry(attempts):
    if len(attempts) >= MAX_ATTEMPTS:
        return False
    m = multiplier(attempts)
    if m is None or m >= MAX_MULT:
        return False
    last = attempts[-1]
    if last.outcome in {"timeout", "cancelled"}:
        return False  # timeouts retry poorly on contended free lanes
    if last.keep_error_in_context and last.tokens_in > 2500:
        return False  # snowball already started
    return last.outcome == "tool_error"

Notice the timeout rule. A tool error sometimes pays for a retry if you strip the huge stderr and keep a one-line cause. A timeout on free capacity usually means the lane is busy or preempted. Retrying a timeout on the same lane is how you donate more queue time to a job that already told you it cannot finish.

Strip before you loop. That is the cheapest retry you will ever buy. Keep the command, the exit code, and one diagnostic line. Drop the rest of the compiler novel. If you cannot strip, you should not retry, because the next prompt is a worse version of the last one. You would not paste three failed CI logs into a new ticket and call that “context.” Do not let the agent do it either.

def compact_tool_error(stderr: str, exit_code: int, cmd: str) -> str:
    lines = [ln.strip() for ln in stderr.splitlines() if ln.strip()]
    last = lines[-1] if lines else "(no stderr)"
    return f"{cmd} exited {exit_code}: {last[:240]}"

Use free capacity when the job is a rehearsal: a prompt you are still shaping, a script you are still wiring, a one-off question you can drop. Kill it when the multiplier climbs. Do not promote that lane to “the way we ship the patch.”

Refuse the free lane when the work has a human waiting on it. Refuse it when the loop can mutate a repo or an environment. Refuse it when the prompt is already large, when the last failure was a timeout or a cancel, or when you cannot log tokens per attempt. In those cases the cheap lane is the expensive decision. The retry multiplier turns a gift into a stall, and the stall still burns context.

Think of free capacity as a bus that stops when it is full. Committed work is a seat you bought. You do not put a deploy-blocking test fix on the bus and then act surprised when it gets bounced and you buy three more tickets trying to arrive on time. Spot is for experiments you can abandon. A retry loop is a promise that you will not abandon. Those two facts do not share a lane.

If you need a box to practice the ledger itself, MonkeyCode’s free model access and free server option are enough to run the accounting script and a short, capped rehearsal. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That is the right size of bet: learn the multiplier on a scratch box. It is the wrong size of bet for an unattended retry loop against a live branch.

This ledger does not know a vendor’s billing granularity. Some meters round. Some count cached prefixes differently. Some charge for tool payloads you never see in the chat UI. If you cannot map a row in the log to a row on the invoice, treat the multiplier as a lower bound. The sample rows are not a benchmark. Do not cite 4.5x, 24 seconds, or a threshold of 3.0 as measured results from a fleet. Set the cuts from your own traces. A research spike and a CI fixer are not the same job.

The verdict function is conservative on purpose. It will tell you to leave free capacity earlier than a hopeful operator would. That is the point. Hope is not a cost control. A green last attempt can still be a bad buy if the path to green was three transcripts and a queue.

Who should not use this approach: anyone who cannot instrument attempts; anyone running unattended agents against production credentials; anyone who needs a hard SLA and thinks a free lane will provide one; anyone shopping for a model beauty contest. This is ops arithmetic. It will not pick a model for you. It will tell you when to stop looping.

If the job matters, cap the retries, strip the errors, and put the loop on capacity you actually reserved. Free is for learning the shape of the bill. It is not for surviving the loop.

Don't Rehearse a Bill on Gift Capacity

Quinn Li — Tue, 22 Sep 2026 15:23:06 +0000

You cannot price a paid path by running the same prompt on gift capacity. A free queue has a different wait, a different preemption story, and a different retry surface. Treat it as a scratch pad. Rehearse spend only where the invoice will actually land.

That sounds obvious until a tool-calling agent walks in. One user message looks cheap. Then the model asks for a tool, you stuff the JSON back into the transcript, the context grows, and the "same" job is four prompts with a fatter tail.

On gift capacity you shrug. In production that tail is the bill.

Think of it like timing a delivery van in a bicycle lane. You will get a number. It will be a number about bicycles.

The usual failure is not a mysterious model. It is a staging habit. You point the same prompt text at a free server because the string matches, then you treat latency, retries, and token mix as if they transferred. Matching text is not matching economics. Cold start, boarding delay, mid-loop preemption, and the ratio of input tokens to output tokens all move when the iron and the meter diverge. Your canary starts lying the moment those two diverge.

Label the work before you label the model.

Every job needs a lane. exploratory may ride gift capacity while you are still wrong on purpose. rehearsal must hit the endpoint, the token cap, the tool budget, and the concurrency you will ship. committed is user-facing or tied to a ship clock. If you cannot name the lane, you are already in the wrong one. A prompt without a lane is how a sketch inherits a production retry ceiling, and how a production agent inherits a gift queue.

Here is a small gate you can run locally. It is a classifier, not a FinOps platform. It refuses to send rehearsal or committed work at a gift base URL. It also refuses to pretend a character heuristic is a vendor quote. Read the comments in the file before you trust a single integer it prints.

#!/usr/bin/env python3
"""job_lane.py — refuse to rehearse a bill on gift capacity.

Heuristic only. Ceiling, not a quote. Not a vendor invoice.
"""
from __future__ import annotations

import argparse
import json
import os
import sys
from dataclasses import dataclass
from typing import Literal

Lane = Literal["exploratory", "rehearsal", "committed"]

@dataclass
class JobEnvelope:
    job_id: str
    lane: Lane
    prompt: str
    tool_hops: int = 0
    max_tokens: int = 512
    concurrency: int = 1
    retry_ceiling: int = 1
    base_url: str = ""

    def heuristic_input_tokens(self) -> int:
        # char/4 stand-in. Not a tokenizer. Do not invoice from this.
        return max(1, len(self.prompt) // 4)

    def output_ceiling(self) -> int:
        hops = max(1, self.tool_hops + 1)
        return self.max_tokens * hops

def is_gift_endpoint(base_url: str) -> bool:
    marker = os.environ.get("GIFT_BASE_URL", "").rstrip("/")
    url = (base_url or os.environ.get("LLM_BASE_URL", "")).rstrip("/")
    if not url:
        return False
    if marker and url == marker:
        return True
    lowered = url.lower()
    return "gift" in lowered or "free" in lowered

def gate(job: JobEnvelope) -> dict:
    gift = is_gift_endpoint(job.base_url)
    input_tok = job.heuristic_input_tokens()
    out_ceil = job.output_ceiling()
    ceiling = (input_tok + out_ceil) * max(1, job.retry_ceiling)
    allowed, reason = True, "ok"

    if job.lane in ("rehearsal", "committed") and gift:
        allowed, reason = False, "refusing to rehearse or ship on gift capacity"
    elif job.lane == "committed" and job.retry_ceiling > 2:
        allowed, reason = False, "committed work cannot hide behind a wide retry ceiling"
    elif gift and job.concurrency > 1:
        allowed, reason = False, "concurrency on gift capacity is not a load test"

    return {
        "job_id": job.job_id,
        "lane": job.lane,
        "gift_endpoint": gift,
        "allowed": allowed,
        "reason": reason,
        "heuristic_input_tokens": input_tok,
        "heuristic_output_ceiling": out_ceil,
        "retry_ceiling": job.retry_ceiling,
        "heuristic_token_ceiling": ceiling,
        "note": "ceiling is a tripwire, not a vendor quote",
    }

def simulate_tool_loop(user_msg: str, tool_payloads: list[str]) -> dict:
    """The first message is not the meter. Each tool result is more prompt."""
    transcript = user_msg
    hops = []
    for i, payload in enumerate(tool_payloads, start=1):
        transcript += "\n" + payload
        hops.append({
            "hop": i,
            "transcript_chars": len(transcript),
            "heuristic_tokens": max(1, len(transcript) // 4),
        })
    return {
        "first_message_heuristic_tokens": max(1, len(user_msg) // 4),
        "final_transcript_heuristic_tokens": max(1, len(transcript) // 4),
        "hops": hops,
    }

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--job", required=True)
    parser.add_argument("--simulate-tools", action="store_true")
    args = parser.parse_args()
    raw = json.loads(open(args.job, encoding="utf-8").read())
    job = JobEnvelope(**{k: raw[k] for k in JobEnvelope.__dataclass_fields__ if k in raw})
    result = gate(job)
    if args.simulate_tools:
        tools = raw.get("tool_payloads", [])
        result["tool_loop"] = simulate_tool_loop(job.prompt, tools)
    print(json.dumps(result, indent=2))
    if not result["allowed"]:
        print(result["reason"], file=sys.stderr)
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Save a fixture that pretends to be a rehearsal. Notice the lane and the base URL disagree on purpose. That disagreement is the bug you want CI to catch, not a clever default you want the model to paper over.

{
  "job_id": "agent-rehearsal-014",
  "lane": "rehearsal",
  "prompt": "List open invoices, then draft a dunning note.",
  "tool_hops": 3,
  "max_tokens": 800,
  "concurrency": 4,
  "retry_ceiling": 3,
  "base_url": "https://gift.example.invalid/v1",
  "tool_payloads": [
    "{\"tool\":\"sql\",\"rows\": 42, \"csv\": \"...\"}",
    "{\"tool\":\"crm\",\"notes\": \"prior balance, three emails\"}",
    "{\"tool\":\"policy\",\"tone\": \"firm\"}"
  ]
}

Run the gate twice. First as a reject. Then point base_url at the path you actually pay, drop concurrency to what you will ship, and run it again as a ceiling check. The second run is still not an invoice. It is a tripwire you can fail a build with.

python3 job_lane.py --job rehearsal.json --simulate-tools; echo exit:$?

# expected: exit 2, reason about gift capacity

GIFT_BASE_URL="https://gift.example.invalid/v1" \
LLM_BASE_URL="https://gift.example.invalid/v1" \
python3 job_lane.py --job rehearsal.json; echo exit:$?

The --simulate-tools path is the part most cost dashboards still flatten. Each tool result is not a side channel. It is more prompt. If you only count the first user message, you are counting the first mile of a round trip and calling it the route. Watch the hop list grow. The last hop is where output tokens get expensive, because the model is now reading its own homework plus yours.

That is why gift capacity is the wrong rehearsal. Preemption in the middle of a four-hop tool loop does not just waste time. It often restarts the graph. You already paid in calendar, and on a metered path you may pay in tokens for the partial transcript too. The free path hides that, because you are not looking at an invoice. You are looking at a courtesy.

Queue time still burns calendar. Calendar is what a release train spends. A gift server that boards you in twenty minutes is not a rehearsal of a 400ms p95. It is a different sport with the same jersey. Load tests on that sport will teach finance the wrong lesson, and they will teach you to overfit a queue you will not have on Monday morning.

So split the lanes on purpose, in the job envelope, not in a wiki nobody reads at 2 a.m.

Use gift capacity to be wrong cheaply. Draft the prompt. Throw away the transcript. Confirm the tool schema even parses. Confirm the agent does not need seven hops to ask for the date. Then copy the envelope onto the paid path and run a short rehearsal at production concurrency, with the same max_tokens, the same stop conditions, and the same retry ceiling. Record input tokens, output tokens, tool hops, queue wait, and wall time as separate columns. If you fold them into one duration number, you will optimize the wrong column. Duration can look fine while output tokens quietly eat the month.

For the exploratory lane only, MonkeyCode — an open-source project with free model access and a free server option — can hold draft-and-discard work so you do not burn the paid meter while the schema is still wrong. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Park sketches there, then move rehearsal onto the endpoint you will actually invoice.

A limitation worth saying out loud: this gate does not know your vendor's billing rules. Some meters count killed streams. Some count cached prefixes differently. Some round up. Some bill tool payloads at a different rate than the surrounding prose. The char/4 stand-in will drift on code, on CJK, and on compressed JSON. Use it as a tripwire that fails closed when a lane and an endpoint disagree. Do not use it as a quote you paste into a budget slide.

Who should not put work on gift capacity at all: anything carrying secrets or regulated user data; anything you plan to treat as an availability replica; any load test whose numbers will be shown to finance as a forecast. Gift capacity is not a cluster you own. It is not a capacity reserve. It is a courtesy with a different SLO, which is to say, maybe none. If your agent is the product, the free path is the wrong bet for canaries. If your job is still a sketch, burning the paid meter to find a better system prompt is the other wrong bet.

Keep those two mistakes from sharing a queue. Name the lane. Gate the URL. Count every hop. Then believe the invoice, not the gift.

Killed Work Still Hits the Meter

Quinn Li — Mon, 21 Sep 2026 13:39:28 +0000

Free capacity is a discount on the next token, not a refund on the last one. If the box dies, the queue expires, or the call is cut off, you still spent the prompt. You still spent the partial answer. You still spent the minutes a human waited. The kill switch does not roll the meter back.

That is the part most teams skip when they park production-shaped work on a scavenger lane. They see a zero in the rate column and stop reading. The invoice is not the rate. The invoice is every token that never became a finished, accepted result.

Think of a taxi the dispatcher can recall mid-ride. The fare looks cheap until the car vanishes at the third stoplight. You do not get those kilometers back. You stand on the curb with a half-finished trip and you pay again to start over. Model jobs behave the same way. Input tokens are prepaid. Partial output is often billed. A restart repeats the prefix you already paid for.

So you need a different unit. Stop asking whether the lane is free. Ask how many times this job can die before the discount costs more than paying now. That number is an abort budget. Without it, spare capacity is not a strategy. It is a leak with a friendly name.

Two lanes, one promotion rule

You already know the shape of the work. Drafts, eval sweeps, prompt archaeology, and throwaway summaries can die. The nightly batch that writes a customer-facing changelog cannot. The classifier that gates a merge cannot. The job that holds a lock in your deploy graph cannot.

Put killable work on scavenged capacity. Put must-finish work on capacity you actually control. The rule is boring, which is why it survives contact with a real queue. A scavenger lane is still useful. You want somewhere to burn exploratory tokens without contaminating the committed path.

MonkeyCode's free model access and free server option can sit in that lane: a box for work you are willing to lose. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That is not a reserved machine, and it is not a claim about quota, hardware, model names, or how long a server stays yours. Treat it as spare capacity you might not get back. If the job cannot tolerate a restart, promote it before the first token. If you want a place to park killable drafts off the paying queue, that free lane is there to try. Measure aborts before you trust it.

An abort budget you can run

The following is a labeled example, not a production scheduler and not a benchmark. You supply the paid token rate and the cost of your own time. The function answers one question: after this many kills, is waiting on free capacity still cheaper than paying?

# abort_budget.py
# Proposal: promote a job off scavenged capacity when restart
# waste meets or exceeds the paid alternative.
from dataclasses import dataclass

@dataclass
class Job:
    name: str
    must_finish: bool
    max_aborts: int
    input_tokens: int
    expected_output_tokens: int
    queue_s_expected: float
    engineer_usd_per_hour: float
    paid_usd_per_million: float

def tokens_burned_on_kill(job: Job, partial_output: int) -> int:
    # Prompt is spent. Partial completion is spent.
    # A restart pays the prompt again.
    return job.input_tokens + max(partial_output, 0)

def waste_usd(job: Job, kills: int, avg_partial_out: int) -> float:
    token_waste = kills * tokens_burned_on_kill(job, avg_partial_out)
    token_usd = token_waste * job.paid_usd_per_million / 1_000_000
    hours = kills * job.queue_s_expected / 3600.0
    time_usd = hours * job.engineer_usd_per_hour
    return token_usd + time_usd

def paid_path_usd(job: Job) -> float:
    total = job.input_tokens + job.expected_output_tokens
    return total * job.paid_usd_per_million / 1_000_000

def should_promote(job: Job, kills: int, avg_partial_out: int) -> bool:
    if job.must_finish:
        return True
    if kills >= job.max_aborts:
        return True
    return waste_usd(job, kills, avg_partial_out) >= paid_path_usd(job)

The comparison uses your paid rate as the opportunity cost of tokens you already threw away. That is deliberate. A free prompt you later repeat on a paid model was never free. It was a loan. You repay it with a second prefix and whatever waiting you did the first time.

Wire a test so the rule cannot drift into folklore.

# test_abort_budget.py
from abort_budget import Job, should_promote, waste_usd, paid_path_usd

def sample_job(**kwargs) -> Job:
    base = dict(
        name="draft-summary",
        must_finish=False,
        max_aborts=2,
        input_tokens=8_000,
        expected_output_tokens=1_200,
        queue_s_expected=180.0,
        engineer_usd_per_hour=75.0,
        paid_usd_per_million=3.0,
    )
    base.update(kwargs)
    return Job(**base)

def test_must_finish_promotes_immediately():
    job = sample_job(must_finish=True, name="release-notes")
    assert should_promote(job, kills=0, avg_partial_out=0) is True

def test_killable_job_stays_until_budget_breaks():
    job = sample_job()
    assert should_promote(job, kills=0, avg_partial_out=400) is False
    assert should_promote(job, kills=2, avg_partial_out=400) is True

def test_long_queue_promotes_before_token_waste_does():
    job = sample_job(queue_s_expected=2400.0, max_aborts=9)
    # Forty minutes of blocked waiting, twice, is not a discount.
    assert waste_usd(job, 2, 0) > paid_path_usd(job)
    assert should_promote(job, kills=2, avg_partial_out=0) is True

Those dollar figures are illustrations you must replace. They are not product prices, not market rates, and not a claim about your bill. If you do not know the paid rate, stop and export one day of usage from the provider you actually use. A budget with a made-up rate is a story.

python -m pip install pytest
pytest -q test_abort_budget.py

You can also poke the function from a shell without standing up a framework. Keep the numbers in environment variables so a teammate can disagree with the rate instead of editing the rule.

python - <<'PY'
from abort_budget import Job, should_promote, waste_usd, paid_path_usd
job = Job(
    name="eval-sweep",
    must_finish=False,
    max_aborts=3,
    input_tokens=12_000,
    expected_output_tokens=800,
    queue_s_expected=90.0,
    engineer_usd_per_hour=75.0,
    paid_usd_per_million=3.0,
)
print("paid_path", round(paid_path_usd(job), 6))
print("waste_after_2", round(waste_usd(job, 2, 300), 6))
print("promote", should_promote(job, kills=2, avg_partial_out=300))
PY

Read the incomplete calls

The budget only works if you can see kills. Most traces hide them under a generic timeout. You want one line per attempt: job id, attempt number, tokens in, tokens out, terminal state. Terminal state is the whole point. complete, timeout, preempted, and cancelled are different invoices.

# attempt_log.py
# Proposal: parse JSONL attempts and split spent tokens into
# finished work versus killed work.
import json
import sys

def summarize(path: str) -> None:
    spent = 0
    finished = 0
    killed = 0
    with open(path, encoding="utf-8") as handle:
        for raw in handle:
            row = json.loads(raw)
            used = int(row["tokens_in"]) + int(row["tokens_out"])
            spent += used
            if row["state"] == "complete":
                finished += used
            else:
                killed += used
    print(f"spent_tokens={spent}")
    print(f"finished_tokens={finished}")
    print(f"killed_tokens={killed}")
    if spent:
        print(f"killed_ratio={killed / spent:.3f}")

if __name__ == "__main__":
    summarize(sys.argv[1])

Start with a fixture so the parser has something honest to chew on.

cat > attempts.jsonl <<'EOF'
{"id":"draft-summary","attempt":1,"tokens_in":8000,"tokens_out":410,"state":"preempted"}
{"id":"draft-summary","attempt":2,"tokens_in":8000,"tokens_out":0,"state":"timeout"}
{"id":"draft-summary","attempt":3,"tokens_in":8000,"tokens_out":1180,"state":"complete"}
EOF
python attempt_log.py attempts.jsonl

If killed_ratio climbs while must_finish jobs are still on the scavenger lane, you are not saving. You are rehearsing the same prompt. That is cousin to a retry storm, except the trigger is the machine going away rather than the model returning an error. The fix is promotion, not a longer timeout. A timeout is how you donate a larger partial completion to the killed column.

Watch the shape of the waste. A huge prompt with a tiny completion is a bad scavenger candidate, because every kill repeats the expensive half. A short prompt with a long, optional draft is a better one. The log will tell you which job you actually have. Your intuition about "cheap experiments" will not.

When free capacity is the wrong bet

Use the scavenger lane when the result is optional, the prompt is cheap to replay, and nobody is blocked on the output. Eval harnesses, prompt diffs, and personal drafts fit. A free server is a fine place to learn whether a prompt is even worth paying for.

Keep committed work off that lane. If the output enters a customer path, a compliance record, or a deploy gate, the job already has an SLO whether you wrote it down or not. If a human is sitting in a meeting waiting on the result, their time will beat a token discount in a single missed slot. If the prompt is large and the completion is small, a kill repeats the costly half and the discount evaporates on the first preempt.

Do not use spare capacity as capacity planning. A box that can vanish is not a headcount substitute and not an SLO. If you need the job tomorrow morning, you already know the answer. Pay, or cut the scope until the job can die without consequences.

The approach also fails if you cannot attribute tokens per attempt. Shared keys, one giant chat session, and dashboards that hide usage behind a monthly total will lie to you. Fix the log first. Then set max_aborts. Then promote.

Limitations are blunt. This model ignores cache hits, ignores provider-side retries you cannot see, and ignores the chance that the paid path is congested too. It assumes you can move a job between lanes without rewriting it. If your orchestration cannot redirect a single task, the function is a comment, not a control. It also assumes the scavenger lane still costs you nothing per token, which may not remain true, and which you should re-read in the product's own terms rather than in a post.

You should not adopt this if you need a guarantee. You should not adopt it if you cannot kill the job. You should not adopt it if the work is someone else's data you are not allowed to send to a shared box. You should not adopt it if the only number you track is requests, because requests are how incomplete work hides.

The core move is small. Tag the job. Cap the kills. Promote when the waste crosses the paid path. Everything else is decoration on a meter that already started.

Free Capacity Is Spot. Committed Work Is Not.

Quinn Li — Sun, 20 Sep 2026 11:38:53 +0000

You should not put a customer-facing job on free model access or a free server. The invoice looks kind. The calendar does not. Free capacity behaves like a spot instance: it is real, it is useful, and it can vanish or stall without warning the moment you needed a percentile, not an average.

That is the whole note. The rest is how you stop mixing the two lanes.

Cloud engineers already learned this with interruptible VMs. You compile on them. You crawl on them. You do not put checkout on them. Language-model work forgot the lesson because the unit of waste is quieter. A preempted VM leaves a hole in a fleet dashboard. A stalled generation leaves a half-built answer, a warm context you cannot reuse, and a human still waiting in the same ticket.

Name the work before the first token. Speculative jobs are searches, prompt comparisons, throwaway extractions, overnight summaries nobody will page you about. Interruption is annoying. It is not a business event. Committed jobs are different. A user is blocked, a pipeline gate is red, or a room is waiting on a number. From that side of the glass, retries are still the same job. The scarce resource is not tokens. It is the remaining minutes before the room goes cold.

You already know the failure mode. The job “looks small,” so you park it on the free lane. The queue is polite for twenty minutes. Then the worker thrashes, the context is rebuilt from scratch, and you save a few cents by missing a release train. Cheap on the bill. Dear on the week. Free capacity is the wrong bet the instant someone else’s clock is attached to the result.

The hidden line item is partial work. Free pools are not obligated to finish. You pay in tokens that produced no success, in queue time that aged the request, and in the cognitive tax of checking whether the job is slow or dead. None of that prints on a vendor invoice. It still hits the same budget: attention. A lane that dies at 90% complete is not a discount. It is an unbilled retry factory.

So you need a router, not a vibe. Classify the job before you spend context. Record waste when the free lane abandons you. Promote or fail fast. Do not “see how it goes” on a path that has a deadline. The snippet below is a proposal you can run locally. It is not a vendor SLO, and it does not pretend to know how any free pool schedules work.

from dataclasses import dataclass
from enum import Enum


class Lane(str, Enum):
    FREE = "free"
    GUARANTEED = "guaranteed"


class JobClass(str, Enum):
    SPECULATIVE = "speculative"
    COMMITTED = "committed"


@dataclass
class Job:
    name: str
    job_class: JobClass
    token_budget: int
    wall_deadline_s: float
    customer_blocked: bool


@dataclass
class Attempt:
    lane: Lane
    tokens_used: int
    elapsed_s: float
    completed: bool
    preempted: bool


class WasteLedger:
    def __init__(self) -> None:
        self.wasted_tokens = 0
        self.completed_tokens = 0
        self.preemptions = 0

    def record(self, attempt: Attempt) -> None:
        if attempt.completed:
            self.completed_tokens += attempt.tokens_used
            return
        self.wasted_tokens += attempt.tokens_used
        if attempt.preempted:
            self.preemptions += 1

    def waste_ratio(self) -> float:
        total = self.wasted_tokens + self.completed_tokens
        return 0.0 if total == 0 else self.wasted_tokens / total


def route(job: Job) -> Lane:
    # Committed work never starts on spot-like capacity.
    if job.job_class == JobClass.COMMITTED or job.customer_blocked:
        return Lane.GUARANTEED
    if job.wall_deadline_s < 30:
        return Lane.GUARANTEED
    return Lane.FREE


def should_stop_using_free(ledger: WasteLedger) -> bool:
    # Gray jobs only. If free keeps dying, stop arguing with it.
    return ledger.waste_ratio() > 0.35 and ledger.preemptions >= 2

Read the policy the way you would read a spot-fleet rule. Speculative jobs may sit on free. Committed jobs never start there. A short wall clock is a committed signal even if you labeled the ticket “research.” If you insist on a hybrid for a gray job, the waste ledger is the adult in the room. A free attempt that burns 8k tokens and then disappears did not save you 8k tokens. It borrowed them from the next guaranteed run.

Wire a heartbeat if your runner can. A free server that stops heartbeating is a preemption, not a thoughtful model. Promote the job or fail it. Do not wait for the HTTP client timeout to teach you the same fact ten minutes later. You can exercise the classifier without calling any network at all:

python - <<'PY'
from cost_router import Job, JobClass, Lane, route, WasteLedger, Attempt, should_stop_using_free

release = Job("release-notes", JobClass.COMMITTED, 4000, 120, True)
explore = Job("prompt-bakeoff", JobClass.SPECULATIVE, 12000, 1800, False)
assert route(release) is Lane.GUARANTEED
assert route(explore) is Lane.FREE

ledger = WasteLedger()
ledger.record(Attempt(Lane.FREE, 8100, 94, False, True))
ledger.record(Attempt(Lane.FREE, 6400, 71, False, True))
assert should_stop_using_free(ledger) is True
print("waste_ratio", round(ledger.waste_ratio(), 3))
print("committed_lane", route(release).value)
PY

Keep a one-line log you can grep when a “cheap” afternoon turns expensive. Tokens without a success are not a research souvenir. They are a failed attempt, same as a killed container.

printf '%s\n' \
  'ts=2026-09-20T18:04:11Z job=prompt-bakeoff lane=free tokens=8100 completed=0 preempted=1' \
  'ts=2026-09-20T18:06:02Z job=release-notes lane=guaranteed tokens=1900 completed=1 preempted=0'

Add a tiny test so the rule cannot drift into “just this once.” Once is how committed traffic leaks onto spot.

from cost_router import Job, JobClass, Lane, route


def test_blocked_user_never_starts_on_free():
    job = Job("support-reply", JobClass.SPECULATIVE, 800, 600, True)
    assert route(job) is Lane.GUARANTEED


def test_true_speculation_may_use_free():
    job = Job("corpus-skim", JobClass.SPECULATIVE, 20000, 7200, False)
    assert route(job) is Lane.FREE

Run pytest -q in the same directory. If that test ever fails because someone “needed the free lane for a demo,” you have found the incident before production finds it. Demos with an audience are committed work. Treat them that way.

Where does a free product actually belong in this picture? On the speculative lane, and only there. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is a reasonable place to park bakeoffs, schema guesses, and overnight skims—the jobs you can afford to lose. It is the wrong place to park a customer-blocked reply or a release-train gate. If you try it, keep the router above in front of the client so curiosity and commitments do not share a queue.

The numbers will lie if you let them. Waste ratio ignores human time. A free job that completes after forty minutes can still be a miss against a ten-minute SLA, and a guaranteed job that spends more tokens can still be cheaper in calendar cost. Do not turn the ledger into a beauty contest between unit prices. Compare finished work against the clock that actually matters.

This policy is the wrong tool if you have no customer, no gate, and no clock. A weekend toy can live entirely on free capacity. A research corpus that may be thrown away tomorrow can too. Do not build a two-lane router for a script you run twice. The overhead of classification only pays for itself when mixed traffic would otherwise hide a stall inside a “savings” story.

Limitations are blunt. You cannot observe another operator’s preemption policy from your laptop. Heartbeats can lie. Token counts can lag. A job labeled speculative on Monday becomes committed on Tuesday when a stakeholder starts waiting. Re-classify at enqueue time, not at idea time. And never use free-lane success on a quiet afternoon as evidence it will hold a percentile on a busy one. Spot markets look generous until they do not.

If you take nothing else: invoice cost and delivery cost are different currencies. Convert before you route. Park speculation on free. Buy the path that ships.

Time in Queue Is Not Free

Quinn Li — Sat, 19 Sep 2026 09:44:30 +0000

Free capacity is cheap only when nobody is waiting on it. The moment a job sits in a shared queue, you start paying in wall-clock time: CI runners stay reserved, reviewers refresh a spinner, and retries take another ticket in the same line. The token meter can read zero while the pipeline gets more expensive.

A free lane is a courtesy checkout. It is not a reserved counter. If twelve other carts are ahead of you, the unpaid register still costs you the afternoon. Shared inference works the same way. Free model access and a free server absorb burst by making you wait, not by printing extra workers on demand.

You already track tokens. Track the wait.

The useful split is not prompt versus completion. It is queued versus generating versus retrying. Queue wait is the gap between "I submitted this" and "a worker actually started." Generation time is the model doing work. Retry time is the same job buying another ticket. If wait dominates, you do not have a model-cost problem. You have a scheduling problem, and free capacity is often the wrong bet.

The block below is a proposed instrumentation pattern, not a production war story. It does not assume a vendor SDK. You wrap the client you already use and write one JSON line per attempt, including failures.

# Proposed span schema. Unexecuted example — wire it to your client.
from __future__ import annotations

from dataclasses import asdict, dataclass
from time import time
import json
import os
from typing import Callable, TypeVar

T = TypeVar("T")

@dataclass
class InferenceSpan:
    job: str
    lane: str  # "free" | "paid" | "local"
    queued_at: float
    started_at: float | None = None
    first_token_at: float | None = None
    finished_at: float | None = None
    tokens_in: int = 0
    tokens_out: int = 0
    retries: int = 0
    ok: bool = False
    error: str | None = None

    def wait_s(self) -> float:
        if self.started_at is None:
            return max(0.0, (self.finished_at or time()) - self.queued_at)
        return max(0.0, self.started_at - self.queued_at)

    def generate_s(self) -> float:
        if self.started_at is None or self.finished_at is None:
            return 0.0
        return max(0.0, self.finished_at - self.started_at)

    def wait_ratio(self) -> float:
        total = self.wait_s() + self.generate_s()
        return 0.0 if total <= 0 else self.wait_s() / total


def timed_call(job: str, lane: str, send: Callable[[], T], span_path: str) -> T:
    span = InferenceSpan(job=job, lane=lane, queued_at=time())
    try:
        span.started_at = time()  # move this to first byte if your client can
        result = send()
        span.first_token_at = span.first_token_at or span.started_at
        span.finished_at = time()
        span.ok = True
        return result
    except Exception as exc:
        span.finished_at = time()
        span.error = type(exc).__name__
        raise
    finally:
        with open(span_path, "a", encoding="utf-8") as fh:
            fh.write(json.dumps(asdict(span)) + "\n")
        os.environ["LAST_WAIT_RATIO"] = f"{span.wait_ratio():.3f}"

Emit the span when the call finishes, even on failure. Especially on failure. A timeout that never started is still a wait. A throttle that bounced you to the tail of the queue is a wait plus a retry. If you only log successful completions, you hide the expensive part of the free lane.

Once you have a day of spans, fold them. Do not start with averages. Averages hide the one job that blocked a person. Group by job name and lane, then look at p50 and p95 wait, generate time, wait_ratio, and retry count. The script below is a proposal you can run against newline-delimited logs.

# Proposed log fold. Unexecuted example.
import json, statistics, sys
from collections import defaultdict

rows = defaultdict(list)
for line in sys.stdin:
    s = json.loads(line)
    wait = 0.0 if s["started_at"] is None else max(0.0, s["started_at"] - s["queued_at"])
    gen = 0.0 if not s["started_at"] or not s["finished_at"] else max(0.0, s["finished_at"] - s["started_at"])
    total = wait + gen
    ratio = 0.0 if total <= 0 else wait / total
    rows[(s["job"], s["lane"])].append((wait, gen, ratio, int(s.get("retries") or 0), bool(s["ok"])))

def pct(xs, p):
    if not xs:
        return 0.0
    xs = sorted(xs)
    i = min(len(xs) - 1, max(0, round((p / 100) * (len(xs) - 1))))
    return xs[i]

print("job,lane,n,p50_wait_s,p95_wait_s,p50_ratio,retries,ok_rate")
for (job, lane), items in sorted(rows.items()):
    waits, gens, ratios, retries, oks = zip(*items)
    ok_rate = sum(oks) / len(oks)
    print(f"{job},{lane},{len(items)},{pct(waits,50):.2f},{pct(waits,95):.2f},{pct(ratios,50):.2f},{sum(retries)},{ok_rate:.2f}")

Read wait_ratio as a tax rate. If a nightly batch sits for eight minutes and generates for forty seconds, the tax is ugly on paper and still fine in practice, because nothing else was blocked. If a pull-request summary sits for eight minutes while the author is in the review thread, the tax is the author's afternoon. Same queue. Different bill.

Translate that into money without inventing a vendor price. You already know your CI rate and a rough loaded cost for an engineer minute. Wait cost is wait_seconds times those rates. Token savings is the tokens you did not send to a paid endpoint. When the job is on a blocked path and wait_cost exceeds token_savings, leave the free lane. When the job can drift to a later hour, stay. That comparison is the whole policy. You do not need a platform committee to run it.

# Proposed decision, not a benchmark. Fill in your own rates.
wait_cost = wait_seconds * (ci_usd_per_minute / 60 + engineer_usd_per_hour / 3600)
token_savings = tokens * paid_usd_per_token
blocked = job_holds_human_or_release_gate

if blocked and wait_cost > token_savings:
    move job off the free lane
elif not blocked:
    keep the free lane and give the job a deadline you can miss

Retries make the comparison worse, because each retry is a fresh ticket. A cheap job that times out in a long queue does not fail once. It fails, waits, fails, waits. You have seen this in build systems: a flaky test is not expensive because the assertion is hard. It is expensive because the queue keeps taking it back. Treat model retries the same way. Cap them on the free lane. If you need reliability, you need a lane with admission control, not more optimism.

Here is a small gate you can drop into CI. It reads the span file for jobs that block merge, and it fails the step when wait_ratio on the free lane crosses a threshold you chose. The numbers are placeholders. Replace them after you look at a week of your own spans.

#!/usr/bin/env bash
# Proposed CI gate. Unexecuted example.
set -euo pipefail
SPAN_FILE="${SPAN_FILE:-.inference-spans.jsonl}"
MAX_RATIO="${MAX_RATIO:-0.70}"
BLOCKING_JOBS="${BLOCKING_JOBS:-pr-summary ci-review-bot}"

if [[ ! -f "$SPAN_FILE" ]]; then
  echo "no span file; refusing to claim the free lane was cheap" >&2
  exit 2
fi

python - "$SPAN_FILE" "$MAX_RATIO" $BLOCKING_JOBS <<'PY'
import json, sys
path, max_ratio, *jobs = sys.argv[1:]
blocking = set(jobs)
max_ratio = float(max_ratio)
bad = []
with open(path, encoding="utf-8") as fh:
    for line in fh:
        s = json.loads(line)
        if s.get("lane") != "free" or s.get("job") not in blocking:
            continue
        started, queued, finished = s.get("started_at"), s["queued_at"], s.get("finished_at")
        wait = (finished or started or queued) - queued if started is None else started - queued
        gen = 0.0 if not started or not finished else finished - started
        total = max(0.0, wait) + max(0.0, gen)
        ratio = 0.0 if total <= 0 else wait / total
        if ratio > max_ratio:
            bad.append((s["job"], ratio, wait))
if bad:
    for job, ratio, wait in bad:
        print(f"free lane too slow for {job}: wait_ratio={ratio:.2f} wait_s={wait:.1f}")
    sys.exit(1)
print("free lane wait_ratio within budget for blocking jobs")
PY

That gate is not a performance test of a model. It is a policy test of a lane. You are asking a simple question: did this work wait so long that "free" became a fiction? If yes, reroute that job. Keep the free lane for work that can be late.

If you need a place to park the late work, MonkeyCode currently offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use that overflow the same way you would use any other unpaid shared lane: batch it, label the span, and keep it off the critical path. This article does not attach model names, quotas, hardware, or uptime claims, because those change and your span file is the only meter that matters.

A practical split looks like this. Overnight evals, prompt diffs you can rerun, log summarization, and backfills can live on free model access or a free server. Anything a person stares at, anything a pager can fire on, and anything that holds a deploy should not. You can move a job between those two columns without rewriting the product. You change the lane label in the span and the endpoint the wrapper calls.

Limitations follow from the shared nature of the lane. Free capacity has no promise of fairness. Your wait_ratio today is not a contract for next Tuesday. Logs will lie if your client reports "start" at enqueue instead of at first byte. If you cannot tell those timestamps apart, you will blame the model for queueing. Do not put secrets, customer transcripts, or regulated records on a free shared server just because the token meter is quiet. Cost is not the same as permission.

Skip this approach if you already run dedicated capacity and your queues are empty. You are solving the wrong problem. Skip it if your product is interactive chat with a latency SLO; do not A/B the SLO against a free lane "to see." You already know the answer. Skip it if you cannot add timestamps to the client. A policy without spans is a slogan.

The habit is small. Write queued_at. Write started_at. Write the ratio. Then decide whether the job may wait. Free inference is a tool for work that can be late. It is a bad hiding place for work that cannot. If you try an overflow lane, export wait_ratio before you move a merge job onto it.

Give Free Capacity a Deadline

Quinn Li — Fri, 18 Sep 2026 08:04:18 +0000

Give free capacity a deadline, or it will spend yours. A token counter cannot tell you that the job sat in a queue until the deploy window closed. You need a second meter, and you need that meter to be allowed to kill the work.

Tokens are a quantity. A release is a time. Those units do not convert. Teams that only watch spend dashboards learn this on a Thursday afternoon, when the cheap path is still “running” and the humans have already lost the day.

Think of a grocery express lane that is free and unstaffed. You do not pay at the register. You pay with the melting ice cream. Interactive coding, review replies, and anything with a person waiting on the other side of the cursor is ice cream. Overnight evals can wait in that lane. The mistake is mixing them because both happen to call a model.

You already know the token clock. Prompt tokens in, completion tokens out, maybe a retry multiplier if you are honest. The calendar clock is cruder and more expensive: enqueue time, first-byte time, done time, and the wall time you promised a human. Free model access and a free server option move cost off the token clock. They do not stop the calendar.

If you are exploring that pair, treat it as opportunistic capacity, not as an SLA. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one place that currently offers free model access and a free server option. The wrapper below does not depend on it. Point the same timer at any endpoint you already run.

Here is the operational claim, stated plainly. You may use spare capacity when the job can die without breaking a promise. You may not use it when a person, a deploy, or a customer ticket is the clock. Hope is not a queue policy. A deadline is.

The proposed control loop is small. Stamp the job when you intend to start, not when the model first speaks. Separate wait from generation. If wait exceeds the budget, abort before you generate a novel. If generation exceeds the budget, abort even if the answer is mid-sentence. Log both clocks in one JSON line so tomorrow-you can argue from a file instead of a feeling.

The script below is an unexecuted example. It talks to whatever URL you put in INFER_URL. Dry-run mode sleeps so you can rehearse the kill switch without credentials or a live model.

#!/usr/bin/env python3
"""wallclock_budget.py — proposed timer for any HTTP inference call."""
import argparse, json, os, sys, time, urllib.error, urllib.request

def emit(event, **fields):
    row = {"event": event, "ts": time.time(), **fields}
    sys.stdout.write(json.dumps(row, separators=(',', ':')) + "\n")
    sys.stdout.flush()

def post(url, body, timeout):
    req = urllib.request.Request(
        url,
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return resp.status, json.loads(resp.read().decode() or "{}")

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--wait-budget-s", type=float, default=20)
    p.add_argument("--total-budget-s", type=float, default=60)
    p.add_argument("--dry-run", action="store_true")
    p.add_argument("--simulate-wait-s", type=float, default=0)
    p.add_argument("--prompt", default="Summarize the failing test in one paragraph.")
    args = p.parse_args()
    url = os.environ.get("INFER_URL", "http://127.0.0.1:8080/v1/complete")
    t0 = time.monotonic()
    emit("enqueued", wait_budget_s=args.wait_budget_s, total_budget_s=args.total_budget_s)

    if args.simulate_wait_s:
        time.sleep(args.simulate_wait_s)
    waited = time.monotonic() - t0
    emit("first_attempt", wait_ms=int(waited * 1000))
    if waited > args.wait_budget_s:
        emit("aborted", reason="wait_budget", wait_ms=int(waited * 1000))
        return 2

    remaining = max(0.1, args.total_budget_s - waited)
    try:
        if args.dry_run:
            time.sleep(min(0.2, remaining))
            status, payload = 200, {"dry_run": True, "text": ""}
        else:
            status, payload = post(url, {"prompt": args.prompt}, timeout=remaining)
    except urllib.error.URLError as exc:
        emit("aborted", reason="transport", error=str(exc), wait_ms=int(waited * 1000))
        return 3

    total = time.monotonic() - t0
    gen_ms = int((total - waited) * 1000)
    if total > args.total_budget_s:
        emit("aborted", reason="total_budget", wait_ms=int(waited * 1000),
             gen_ms=gen_ms, http_status=status)
        return 2
    emit("completed", wait_ms=int(waited * 1000), gen_ms=gen_ms,
         total_ms=int(total * 1000), http_status=status,
         chars=len(json.dumps(payload)))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Rehearse the abort on your laptop before you aim it at a shared box.

chmod +x wallclock_budget.py
python3 wallclock_budget.py --dry-run --simulate-wait-s 25 --wait-budget-s 20
echo exit:$?
python3 wallclock_budget.py --dry-run --simulate-wait-s 2 --wait-budget-s 20 --total-budget-s 60

The first command should exit 2 and print aborted with wait_budget. That is the point. You wanted the failure in twenty seconds, not a silent wait that turns into an unreviewed diff at 6 p.m. The second command should complete. Same prompt, same “free” path, different relationship to the calendar.

When you have a real endpoint, keep the logs. One line per attempt is enough to see whether you are paying in queue or in generation.

export INFER_URL='http://127.0.0.1:8080/v1/complete'
python3 wallclock_budget.py --wait-budget-s 15 --total-budget-s 45 \
  --prompt 'List the three failing assertions and nothing else.' \
  | tee -a infer-clock.jsonl

python3 - <<'PY'
import json
waits, gens, aborts = [], [], 0
for line in open("infer-clock.jsonl"):
    row = json.loads(line)
    if row.get("event") == "aborted":
        aborts += 1
    if "wait_ms" in row and row.get("event") in {"completed", "aborted"}:
        waits.append(row["wait_ms"])
    if row.get("event") == "completed":
        gens.append(row.get("gen_ms", 0))
print("aborts", aborts)
print("p50_wait_ms", sorted(waits)[len(waits)//2] if waits else None)
print("p50_gen_ms", sorted(gens)[len(gens)//2] if gens else None)
PY

Read that file like an invoice with two columns. If wait dominates, you did not get a cheap model. You got a slow ticket counter. If generation dominates and you still miss the window, your prompt is too fat for the remaining minutes. Either way, the token dashboard was never going to show it.

A curl one-liner is enough when you do not want Python in the loop. It is worse at separating wait from generate, and that is the lesson. Wall time without a split is how “the model was slow” becomes an unexamined story.

START=$(date +%s)
curl -sS -m 45 -H 'Content-Type: application/json' \
  -d '{"prompt":"Return only the stack frame that raised."}' \
  "$INFER_URL" >/tmp/infer.out
END=$(date +%s)
echo elapsed_s:$((END-START))

Use the split when the job can wait. Use the blunt curl timeout when you already know the human is in the loop. Do not put an editor-backed request on a path you would not give a p99 to.

The decision is not ideological. It is a shape. A nightly corpus scan with hours of slack can ride spare capacity, including a free server, because a kill at 2 a.m. costs you a rerun, not a meeting. A “fix this test before standup” request cannot. A customer-facing complete-me box cannot. A regulated workload that needs tenancy and an audit trail cannot. Same model family, different clock.

Job shape	Slack on the calendar	Kill switch	Spare / free path
Overnight eval sweep	Hours	Retry tomorrow	Fit
Prompt regression, no human waiting	Tens of minutes	Abort and narrow the batch	Fit if wait stays inside slack
PR comment, pairing, incident reply	Seconds to a few minutes	Fail closed, switch off spare	Poor fit
User-facing completion	Hard p99	Do not abort into a blank UI	Wrong tool

Notice what the table does not say. It does not say free is fake. It does not say paid is virtuous. It says the calendar is the constraint that tokens cannot express. When slack is large, opportunistic capacity is a rational bet. When slack is a standup, you are betting the meeting.

Limitations are not fine print. This wrapper does not see the provider’s internal queue, so wait_ms starts when your process starts, not when their scheduler notices you. Abort may still consume work the remote side already began; a timeout is not a refund. Clocks drift across machines, so run the timer next to the client, not on a dashboard in another region. Dry-run sleeps are not a load test. A free tier can change shape without updating your script. None of this replaces a reserved endpoint for production traffic.

Skip this approach if you ship interactive inference to other people. Skip it if a missed completion is a safety event. Skip it if your org cannot tolerate preemption, noisy neighbors, or a job that vanishes mid-token. Skip it if you were about to use spare capacity to “move faster” on a deadline you already quoted. Speed you cannot timestamp is not speed.

AI-assisted coding makes the mix-up easier, because the editor feels interactive while the remote job is a batch queue in costume. Calling that engineering does not make the ice cream freeze. Measure wait. Cap the total. Keep deadline work off the unstaffed lane. If you already have a free-capacity path, wrap one real job in the timer this week and read wait_ms before you argue about token price.

The Call Graph Is What You Owe

Quinn Li — Thu, 17 Sep 2026 06:08:51 +0000

You are not paying for a prompt. You are paying for every hop the agent takes after that prompt, including the ones you never wrote down.

A user types “fix the failing test.” That looks like one request. In the runtime it becomes a tree: list files, read three of them, propose a patch, run the suite, read the failure, patch again. Each hop has tokens, wall time, and a chance to fan out. If your budget lives on the first call, the tree ignores you.

Think of a restaurant tab that starts with a coffee and then keeps adding plates because nobody closed the table. The coffee was cheap. The table was not. Dashboards that count HTTP requests stay polite while that table fills up. You need a budget object that travels with the job, not with the client.

This is an ops problem, not a model-quality problem. A stronger model can still wander. A weaker one can still finish if the graph is short. Your control point is the graph: how many hops, how wide the fan-out, and when the wrapper says stop.

A job that carries its own meter

The example below is a local runner, not a live agent. It does not call a network model. Each hop returns a fake token count so you can watch the kill switch trip on purpose. Treat the numbers as fixtures. Do not treat them as a benchmark.

# save as graph_budget.py — worked example, not production telemetry
from __future__ import annotations

from dataclasses import dataclass, field, asdict
import json
import time


@dataclass(frozen=True)
class Hop:
    name: str
    tokens_in: int
    tokens_out: int
    fanout: int
    kind: str  # "explore" | "mutate" | "verify"


@dataclass
class JobBudget:
    max_tokens: int
    max_hops: int
    max_seconds: float
    tokens_used: int = 0
    hops_used: int = 0
    started_at: float = field(default_factory=time.monotonic)
    events: list = field(default_factory=list)

    def _seconds(self) -> float:
        return time.monotonic() - self.started_at

    def decide(self, hop: Hop) -> str:
        next_tokens = self.tokens_used + hop.tokens_in + hop.tokens_out
        next_hops = self.hops_used + max(1, hop.fanout)
        if next_tokens > self.max_tokens or next_hops > self.max_hops:
            return "abort"
        if self._seconds() > self.max_seconds:
            return "abort"
        if hop.kind == "explore" and next_tokens > int(self.max_tokens * 0.6):
            return "skip-explore"
        return "run"

    def commit(self, hop: Hop, decision: str) -> None:
        if decision == "run":
            self.tokens_used += hop.tokens_in + hop.tokens_out
            self.hops_used += max(1, hop.fanout)
        self.events.append(
            {
                "hop": hop.name,
                "kind": hop.kind,
                "decision": decision,
                "tokens_used": self.tokens_used,
                "hops_used": self.hops_used,
                "seconds": round(self._seconds(), 4),
            }
        )


# A compact stand-in for "fix the failing test"
GRAPH = [
    Hop("list_files", 800, 120, 1, "explore"),
    Hop("read_a", 2200, 400, 1, "explore"),
    Hop("read_b", 1800, 350, 1, "explore"),
    Hop("read_c", 2600, 500, 1, "explore"),  # extra read you did not budget
    Hop("draft_patch", 3100, 900, 1, "mutate"),
    Hop("run_tests", 400, 1800, 3, "verify"),  # fan-out: 3 failing files
    Hop("patch_again", 3500, 1100, 1, "mutate"),
]


def run_job(max_tokens: int = 12000, max_hops: int = 8, max_seconds: float = 2.0) -> dict:
    budget = JobBudget(max_tokens=max_tokens, max_hops=max_hops, max_seconds=max_seconds)
    for hop in GRAPH:
        decision = budget.decide(hop)
        budget.commit(hop, decision)
        if decision == "abort":
            break
    return {
        "tokens_used": budget.tokens_used,
        "hops_used": budget.hops_used,
        "aborted": any(e["decision"] == "abort" for e in budget.events),
        "events": budget.events,
    }


if __name__ == "__main__":
    result = run_job()
    print(json.dumps(result, indent=2))

Run it from a shell. You should see the job die on the extra read or on the test fan-out, depending on the caps you pass.

python graph_budget.py
python - <<'PY'
from graph_budget import run_job
loose = run_job(max_tokens=50000, max_hops=20)
tight = run_job(max_tokens=9000, max_hops=6)
assert tight["aborted"] is True
assert tight["tokens_used"] < loose["tokens_used"]
print("tight hops", tight["hops_used"], "loose hops", loose["hops_used"])
PY

The assertion is the point. A looser cap finishes more of the tree. A tighter cap leaves work on the table and also leaves money on the table. You choose that trade in code, before the agent chooses it for you in a retry fog.

Read the JSON as an incident note. When kind stays on explore while tokens_used climbs, the job is still shopping for context. When fanout jumps on verify, one failing assertion just cloned itself. Neither of those events looks dramatic in a per-request chart. Both of them are the invoice.

Route hops, do not bless the whole tree

You can put exploratory hops on a free lane without putting the whole graph there. Listing files and drafting a plan are recoverable if they stall. Applying a patch into CI is not. The wrapper above already has a skip-explore branch. That is the cheap door. Keep the mutate and verify hops behind a hard abort.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option. Those two facts matter here only as a place to run the hops you have already capped in software. They do not flatten the call graph. They do not promise queue time, model names, or a quota I am not going to invent. If you use that free lane, use it for the explore steps your budget still allows, then stop.

A useful rule of thumb, still a rule of thumb: if a hop can be thrown away, it may sit on free capacity. If a hop mutates a branch or gates a merge, it needs a cap you would defend in a postmortem. Free compute is a lane. It is not a contract with your on-call rotation.

Try a second command that pretends the explore phase got greedy. You will watch skip-explore fire before abort, which is the shape you want in staging.

python - <<'PY'
from graph_budget import Hop, JobBudget

budget = JobBudget(max_tokens=10000, max_hops=10, max_seconds=3)
greedy = [
    Hop("repo_dump", 7000, 200, 1, "explore"),
    Hop("more_dump", 4000, 200, 1, "explore"),
    Hop("draft_patch", 2000, 800, 1, "mutate"),
]
for hop in greedy:
    d = budget.decide(hop)
    budget.commit(hop, d)
    print(hop.name, d, budget.tokens_used)
PY

Notice what did not happen. The mutate hop never ran. That is not elegance. That is an adult closing the tab before dessert. Context stuffing feels like diligence when you are staring at a red test. It is how a one-call estimate becomes a tree you cannot explain to finance.

What this does not fix

A circuit breaker on hops will not save a bad spec. If you ask the agent to “make it work” with no file list and no failing assertion, the graph has to go hunting. Hunting is fan-out with a friendly name. Write the failing command into the job record first. Then let the agent propose a patch against that record. The budget object can only bound a search you were willing to bound.

Do not use this pattern for hard real-time work, for customer-facing agents with a latency SLO, or for anything that must finish because a human is blocked on a call. A free server and a free model path are the wrong bet when the cost of waiting exceeds the cost of the tokens. They are also the wrong bet when a skipped explore hop would have prevented a bad mutate. Caps create false confidence if you never inspect the skip log.

This also does not replace tracing. If two CI jobs spawn the same graph against the same SHA, you will pay twice and the breaker on each job will still say “within budget.” Dedup lives one layer up, in the workflow that decides whether the agent should run at all. Pair the meter with a job key. Without that key, you are just dying cheaper, more often.

Keep the title of the work honest in your own runbooks. You did not ship “an AI code fix.” You shipped a bounded graph with an abort. That sentence survives if every product name in this article is deleted. The product, if you use it, is only a lane for the hops you already decided were disposable.

If you want to feel the abort on a throwaway repo before you wire it into CI, run the scripts above, then point only the explore hops at MonkeyCode’s free model access or free server and leave mutate/verify behind the same cap. One experiment is enough. The graph will tell you whether the next hop is work or just another plate on the table.

Retries Are How Cheap Jobs Get Expensive

Quinn Li — Wed, 16 Sep 2026 04:18:25 +0000

Retries are how cheap jobs get expensive. A failed generation does not refund the tokens you already burned, and it does not restore your place in line. It opens a second job that looks identical in a per-request dashboard and twice as costly in any honest ledger.

You already know the first attempt has a price. The trap is treating the second attempt as a mulligan. In shipping, a returned parcel is not free to send again. You pay postage, you pay handling, and you wait behind everyone who shipped once. LLM work behaves the same way once you stop staring at status codes and start staring at attempts.

Per-request charts hide that. One green check after four failures still plots as a single success. The queue saw four arrivals. The tokenizer saw four growing prompts. Your bill saw four completions, including the ones you threw away. If you only alert on error rate, a retry storm looks like resilience. It is spend with a friendly name.

Think of an agent loop as a multiplier, not a safety net. The first call ships a system prompt, a file excerpt, and a tool schema. The second call ships all of that plus the failed tool trace. The third call ships the argument about the second call. Context does not reset when hope resets. Each retry is a fatter invoice re-entering the same contested lane.

That is why free capacity is often the wrong bet after the first miss, even when the work is not on a deadline. Free lanes are shared. Shared lanes punish re-entry. You do not merely wait. You wait, then you pay to reconstruct state, then you wait again. The cheap path becomes the long path, and the long path becomes the expensive path because tokens and queue time compound together.

Price the next attempt before you launch it. Not the model. The attempt. Give every job an attempt budget with three hard numbers you can defend: a maximum retry count, a cumulative token ceiling, and a re-queue time ceiling. When any ceiling trips, stop or promote. Do not ask the model if it feels lucky.

The script below is a worked example, not a production benchmark and not a claim about any vendor's quota. Run it locally. Feed it your own traces. Treat the numbers as a method, not a result.

#!/usr/bin/env python3
"""Attempt ledger for LLM jobs. Example only; plug in your tracer."""
from __future__ import annotations

from dataclasses import dataclass, field
from typing import Literal, Optional

Route = Literal["free", "paid"]

@dataclass
class Attempt:
    job_id: str
    n: int
    tokens_in: int
    tokens_out: int
    queue_ms: int
    ok: bool
    reason: str

@dataclass
class Budget:
    max_attempts: int = 3
    max_tokens: int = 24_000
    max_queue_ms: int = 20_000

@dataclass
class Ledger:
    job_id: str
    route: Route
    budget: Budget
    attempts: list[Attempt] = field(default_factory=list)

    def tokens(self) -> int:
        return sum(a.tokens_in + a.tokens_out for a in self.attempts)

    def queue_ms(self) -> int:
        return sum(a.queue_ms for a in self.attempts)

    def next_allowed(self) -> tuple[bool, str]:
        if len(self.attempts) >= self.budget.max_attempts:
            return False, "attempt_cap"
        if self.tokens() >= self.budget.max_tokens:
            return False, "token_cap"
        if self.queue_ms() >= self.budget.max_queue_ms:
            return False, "queue_cap"
        return True, "ok"

    def record(self, tokens_in: int, tokens_out: int, queue_ms: int, ok: bool, reason: str) -> Attempt:
        attempt = Attempt(
            job_id=self.job_id,
            n=len(self.attempts) + 1,
            tokens_in=tokens_in,
            tokens_out=tokens_out,
            queue_ms=queue_ms,
            ok=ok,
            reason=reason,
        )
        self.attempts.append(attempt)
        return attempt

    def promote_or_stop(self) -> str:
        allowed, why = self.next_allowed()
        if allowed:
            return f"retry_on_{self.route}"
        if self.route == "free" and why in {"queue_cap", "attempt_cap"}:
            return "promote_to_paid_once"
        return f"stop:{why}"

def simulate_retry_storm(job_id: str) -> None:
    # Synthetic trace so you can see the multiplier. Replace with real spans.
    growing_prompt = 1800
    ledger = Ledger(job_id=job_id, route="free", budget=Budget())
    for i in range(6):
        decision = ledger.promote_or_stop()
        print(f"pre-attempt decision={decision} tokens={ledger.tokens()} queue_ms={ledger.queue_ms()}")
        if not decision.startswith("retry"):
            print("circuit open")
            return
        queue = 1500 * (i + 1)  # re-entry gets slower on a busy free lane
        out = 400
        ok = False
        ledger.record(growing_prompt, out, queue, ok, reason="tool_schema_mismatch")
        growing_prompt += 350  # failed tool trace sticks to the next prompt
    print("unbounded loop would have continued")

if __name__ == "__main__":
    simulate_retry_storm("codegen-414")

Run it as a dry rehearsal before you wire a tracer.

python3 attempt_ledger.py

You should see the circuit open on the attempt cap or the queue cap, not on the sixth hopeful call. That is the point. The job does not get to negotiate after it has already multiplied. If your real traces never trip the cap, your budget is theater. Tighten it until a noisy tool failure stops itself.

Wire the same object to whatever you already log. You do not need a new platform. You need three fields on every span: attempt_n, tokens_in+tokens_out, and queue_ms. If a vendor hides queue time, record scheduled_at and first_token_at yourself. Subtract. Attach both timestamps to the job id, not to the HTTP request id. Request ids change on retry. Job ids must not.

A small assertion keeps the ledger honest in CI. The test is a contract on accounting, not on model quality.

import unittest
from attempt_ledger import Budget, Ledger

class LedgerContract(unittest.TestCase):
    def test_retry_is_a_new_invoice(self):
        led = Ledger("j1", "free", Budget(max_attempts=2, max_tokens=10_000, max_queue_ms=9_000))
        led.record(2000, 500, 4000, False, "parse_error")
        led.record(2350, 500, 4000, False, "parse_error")
        self.assertEqual(led.tokens(), 5350)
        self.assertEqual(led.queue_ms(), 8000)
        self.assertEqual(led.promote_or_stop(), "stop:attempt_cap")

    def test_free_lane_promotes_on_queue_cap(self):
        led = Ledger("j2", "free", Budget(max_attempts=5, max_tokens=50_000, max_queue_ms=3000))
        led.record(800, 200, 3500, False, "timeout")
        self.assertEqual(led.promote_or_stop(), "promote_to_paid_once")

if __name__ == "__main__":
    unittest.main()

python3 -m unittest attempt_ledger_test.py -v

Notice what the contract refuses to do. It does not ask whether the second answer would have been better. It does not average tokens across successes. It does not bless a free lane because the sticker price is zero. Zero is the price of admission, not the price of re-entry. Re-entry is still work.

Use a routing rule you can explain to someone who does not write agents. First attempt may sit on a free lane if the job is batch, interruptible, and has a tiny prompt. After one failure, look at why. Schema errors and missing files will not heal because you waited. Retrying those on the same lane is how you buy a longer queue for the same bug. Promote once, or stop and fix the tool. Model flakes and truncated outputs can retry, but only inside the budget, and only if the prompt did not grow more than a ratio you set. A doubling transcript is not a retry. It is a different job wearing the same ticket.

That last distinction matters more than model choice. Teams argue about which checkpoint is smarter while the loop quietly restates three files of dead context. You can watch it with a one-liner on a JSONL trace if you already emit token counts.

python3 - <<'PY'
import json, sys
from collections import defaultdict
jobs = defaultdict(list)
for line in sys.stdin:
    row = json.loads(line)
    jobs[row["job_id"]].append(row["tokens_in"] + row["tokens_out"])
for job, series in jobs.items():
    if len(series) < 2:
        continue
    print(job, "attempts", len(series), "first", series[0], "last", series[-1], "sum", sum(series))
PY

Pipe real logs, not hopes. If last is much larger than first, your retry is eating its own history. Cap context on retry the way you cap attempts. Summarize the failure in one paragraph. Do not reship the whole argument. The analogy is packing: you do not tape the damaged box inside a bigger box and call that a fix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a scratch place to run the ledger against a live coding loop, MonkeyCode's free model access and free server option are enough to instrument attempts without standing up your own box. Treat both as capacity you still have to budget. They do not cancel a retry storm. They only make the storm cheaper to observe.

Limitations sit in the open. This method does not estimate quality. A stopped job can be the right cost decision and the wrong product decision. Token counters differ by vendor and by whether they bill cached prefixes; your ledger is only as true as the meter you trust. Queue time on a laptop is not queue time on a shared free lane, so do not publish the sample numbers above as evidence. They exist to show multiplication, not to rank providers. Promotion to a paid lane can thrash if you promote on the first timeout and the paid lane is also busy. Promote once. Then stop.

You should not use this approach if you cannot attach a stable job id across retries. Without that, you will keep paying for the same failure under new request ids and congratulate yourself on throughput. You should not use it for interactive chat where a human is the circuit breaker. You should not use it as an excuse to skip tests. A parse error retried six times is still a parse error. The budget is there to keep the mistake small, not to launder it into a success metric.

Cheap jobs stay cheap when the second attempt has a price tag before it starts. Everything else is a story you tell after the invoice arrives.

Count Tokens per Success, Not per Request

Quinn Li — Mon, 14 Sep 2026 21:50:11 +0000

The number that decides whether a job belongs on free capacity is not tokens per request. It is tokens per successful task. Almost nobody computes the second one, and that blind spot is why a free tier looks cheap for months and then quietly becomes the most expensive line in your pipeline.

A retry is a second invoice for the same deliverable. You would not accept a courier that bills you again every time it misses your door, yet that is what most LLM wrappers do: three attempts, one answer, three times the tokens. The meter only knows what the client sent.

This is getting worse, not better. Agents are the current default pattern, and an agent loop multiplies every retry you already had. A step that retries three times inside a loop that runs eight iterations is not a 3x overrun on that step. It is up to 24x on the part of the pipeline that was supposed to be the cheap part.

Write the meter before you trust the price

The harness below is deliberately boring. It wraps one OpenAI-compatible chat endpoint, classifies retryable failures, sleeps on backoff, and attributes every attempt to a task. Point it at whatever gateway you actually use; the accounting is the point, not the client.

# token_budget.py — count tokens per successful task, not per request.
# Python 3.10+. Only dependency: httpx.
import os, time, random
from dataclasses import dataclass, field
import httpx

BASE_URL = os.environ['LLM_BASE_URL'].rstrip('/')
API_KEY = os.environ.get('LLM_API_KEY', '')
MODEL = os.environ.get('LLM_MODEL', 'default')

RETRYABLE = {408, 409, 429, 500, 502, 503, 504}
MAX_ATTEMPTS = 5

@dataclass
class Result:
    name: str
    ok: bool = False
    attempts: int = 0
    prompt_tokens: int = 0
    completion_tokens: int = 0
    backoff_seconds: float = 0.0
    wall_seconds: float = 0.0
    errors: list = field(default_factory=list)

def one_call(client, prompt, timeout=60.0):
    started = time.monotonic()
    r = client.post(
        f'{BASE_URL}/chat/completions',
        headers={'Authorization': f'Bearer {API_KEY}'},
        json={'model': MODEL, 'messages': [{'role': 'user', 'content': prompt}]},
        timeout=timeout,
    )
    elapsed = time.monotonic() - started
    if r.status_code in RETRYABLE:
        raise httpx.HTTPStatusError('retryable', request=r.request, response=r)
    r.raise_for_status()
    return r.json().get('usage', {}), elapsed

def run_task(client, name, prompt):
    res = Result(name=name)
    for attempt in range(1, MAX_ATTEMPTS + 1):
        res.attempts = attempt
        try:
            usage, elapsed = one_call(client, prompt)
            res.prompt_tokens += usage.get('prompt_tokens', 0)
            res.completion_tokens += usage.get('completion_tokens', 0)
            res.wall_seconds += elapsed
            res.ok = True
            return res
        except httpx.HTTPStatusError as e:
            status = e.response.status_code
            res.errors.append(status)
            if status not in RETRYABLE:
                return res  # a bug in your request is not a rate limit
            delay = min(2 ** attempt, 30) + random.random()
            res.backoff_seconds += delay
            time.sleep(delay)
        except httpx.TimeoutException:
            res.errors.append('timeout')
            delay = min(2 ** attempt, 30) + random.random()
            res.backoff_seconds += delay
            time.sleep(delay)
    return res

Run it with the environment your own gateway expects:

LLM_BASE_URL=https://your-gateway.example/v1 \
LLM_MODEL=your-model \
LLM_API_KEY=... \
python token_budget.py

Then print the only two ratios that matter:

def report(results):
    spent = sum(r.prompt_tokens + r.completion_tokens for r in results)
    ok = [r for r in results if r.ok]
    attempts = sum(r.attempts for r in results)
    per_success = spent / max(len(ok), 1)
    amplification = attempts / max(len(ok), 1)
    print(f'tasks                : {len(results)}')
    print(f'successful           : {len(ok)}')
    print(f'model attempts       : {attempts}')
    print(f'retry amplification  : {amplification:.2f} attempts per success')
    print(f'tokens per success   : {per_success:.0f}')
    print(f'time slept in backoff: {sum(r.backoff_seconds for r in results):.1f}s')

The shape of the output looks like this — illustrative, not measured, because your endpoint is not mine:

tasks                : 40
successful           : 40
model attempts       : 57
retry amplification  : 1.43 attempts per success
tokens per success   : 3180
time slept in backoff: 214.6s

Read that last line again. You burned three and a half minutes asleep, and the task still succeeded, so nothing on your dashboard fired an alert. Free capacity does not remove that cost. It moves it from your invoice to your wall clock.

Multiply the two amplifiers

The useful mental model is short: effective spend equals base tokens, times attempts per call, times loop iterations. A free endpoint with a 1.4x retry rate and a six-step agent loop is a 8.4x token multiplier on paper you will never see printed anywhere.

There is a second amplifier most teams miss entirely. When a step fails and the agent replans, the upstream context is resent. So the retry does not just repeat one prompt. It repeats the accumulated conversation, which is the largest part of the request. That is where a 1.4x retry rate turns into a 3x bill.

A decision table you can argue with

Situation	Free capacity	Why
Hard deadline measured in hours	Wrong bet	Variance decides the outcome, not the mean, and you cannot sleep through a backoff window
Overnight batch or backfill	Right bet	Backoff costs wall clock nobody is sitting in front of
Latency-sensitive interactive path	Wrong bet	Your p95 is the user experience, and rate limits live in the tail
Throwaway prototyping	Right bet	Failure is a feature; you are buying information cheaply
Job with non-idempotent side effects	Neither until you fix it	A retried write is a duplicated write

That last row deserves attention. Retries are only safe when the operation is idempotent. If a step sends an email, charges a card, or appends to a queue without a deduplication key, retry accounting is the least of your problems.

Where a hosted free option changes the arithmetic

Running the harness against free model access and a free server option is a reasonable way to get real numbers without buying hardware or paying for a bespoke deployment first. MonkeyCode offers both: free model access and a free server option, with an operator-stated free token allowance (currently advertised as 10M tokens — verify the live terms yourself, since allowances change). Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What a free server buys you is the removal of one variable: you are no longer debugging your own box and the queue at the same time. What it does not buy you is a smaller amplification factor. Retries still cost tokens, backoff still costs seconds, and a crowded queue still shows up in your p95. The meter does not care who owns the metal.

So use the free tier as a measurement rig first and a production lane second. Run the batch job there, collect the amplification ratio, then decide whether the deadline-sensitive path has any business being on it. The free allowance is a budget for learning, not a promise about latency.

Who should not take this route

Skip this approach if you have contractual latency targets, if your work has non-idempotent side effects and no deduplication layer, or if you cannot measure at all — a team without the meter will misread variance as a broken vendor. It is also the wrong fit for regulated data unless you have confirmed where prompts are processed, which is a policy question and not a pricing one.

The honest conclusion is unglamorous. Free capacity is not cheap or expensive by nature; it is cheap for work that tolerates retries and expensive for work that does not. Count tokens per success, multiply by your loop depth, and let the number pick the lane.

If you try the harness, bring your own load and your own endpoint. A meter that only agrees with the brochure is not a meter.

The Transcript Is the Bill

Quinn Li — Sun, 13 Sep 2026 20:31:00 +0000

The transcript is the bill. An agent job is not one request, and pricing it as one request is how a free lane disappears without a crash. Every tool turn reprints the prior messages. The model does not remember. You pay to remind it.

You already treat retries as a cost center. This is adjacent and worse. A retry repeats a call. A loop reprints a growing document. The second turn is not twice the first. It is the first plus the tool dump plus the next instruction, stuffed back through the same tokenizer.

That reprint is the unit you should budget. Not “jobs.” Not “agents.” Tokens that reappear because your orchestrator is polite enough to send the whole story again.

The multiplier hides in the polite client

Most agent wrappers look inexpensive in a dashboard because the dashboard counts HTTP calls. One call, one row. The row does not show that call three rehydrated calls one and two. Context is not a cache unless you built a cache. It is a payload you keep mailing to yourself.

Picture a meeting notes app that emails the entire thread every time someone types “ok.” You would not call that one email. You would call it a photocopier with a stamp. Tool-using loops do the same thing, only the paper is tokens and the stamp is your budget.

A tiny heuristic is enough to see it. Four characters per token is wrong in the edges and right enough to scare you.

# proposal: envelope tracer, not a production SDK
from dataclasses import dataclass, field


def approx_tokens(text: str) -> int:
    return max(1, len(text) // 4)


@dataclass
class Turn:
    role: str
    text: str
    tool: str | None = None

    @property
    def tokens(self) -> int:
        return approx_tokens(self.text)


@dataclass
class Envelope:
    cap: int
    spent: int = 0
    rejected: int = 0
    log: list[dict] = field(default_factory=list)

    def charge_prompt(self, transcript: list[Turn], new_output: str) -> bool:
        prompt = sum(t.tokens for t in transcript)
        completion = approx_tokens(new_output)
        cost = prompt + completion
        row = {
            "turn": len(self.log) + 1,
            "prompt_tokens": prompt,
            "completion_tokens": completion,
            "cumulative": self.spent + cost,
            "accepted": self.spent + cost <= self.cap,
        }
        if not row["accepted"]:
            self.rejected += cost
            self.log.append(row)
            return False
        self.spent += cost
        self.log.append(row)
        return True

Run a fake loop against it. Do not wait for a vendor invoice. The shape shows up in thirty lines.

SYSTEM = Turn("system", "You are a repo nurse. Call tools. Do not guess file contents.")
user = Turn("user", "Find why /health returns 500, then patch it.")
transcript = [SYSTEM, user]
env = Envelope(cap=8_000)

tool_blobs = [
    "search: 12 files, 9 false positives, 3 stack traces pasted in full",
    "read: 1_800 lines of middleware, comments included",
    "test: 40 lines of failure, 200 lines of captured logs",
]

for i, blob in enumerate(tool_blobs, start=1):
    thought = f"Turn {i}: I will inspect more context because the last tool was noisy."
    if not env.charge_prompt(transcript, thought):
        print("abort before reprint", env.log[-1])
        break
    transcript.append(Turn("assistant", thought, tool=f"tool_{i}"))
    transcript.append(Turn("tool", blob, tool=f"tool_{i}"))
else:
    print("loop finished inside cap", env.spent)

for row in env.log:
    print(row)

You will watch prompt tokens climb while completion tokens stay small. That is the tell. The model is not “thinking more.” You are mailing a thicker envelope. The third tool result did not cost third-tool money. It cost the whole novel again, plus a paragraph.

Charge the reprint, then decide the lane

Once you have a cumulative column, jobs become comparable. A one-shot completion with a 600-token prompt is a postcard. A four-turn repair agent with the same user sentence is a binder. Free capacity can hold postcards all afternoon. Binders contend. Binders also fail open if you only cap steps.

Step caps lie. Four tools can be cheap if each returns a hash. Four tools can be fatal if each returns a file. Token caps tell the truth because they meter the photocopy, not the choreography.

Put the abort next to the call site, not in a weekly report. A report is a eulogy. An abort is a budget.

def run_turn(env: Envelope, transcript: list[Turn], call_model, tools):
    # labeled example: wire this in front of your real client
    if env.spent >= env.cap:
        raise RuntimeError("envelope already spent")
    prompt_preview = sum(t.tokens for t in transcript)
    if prompt_preview >= env.cap:
        raise RuntimeError("reprint alone exceeds cap")
    output = call_model(transcript)  # your client here
    if not env.charge_prompt(transcript, output.text):
        raise RuntimeError("completion would cross cap")
    transcript.append(Turn("assistant", output.text, tool=output.tool))
    if output.tool:
        blob = tools[output.tool](output.args)
        transcript.append(Turn("tool", blob, tool=output.tool))
    return output

Keep the rejected counter. Discarded completions are still work the machine did. If you stream and the user hits stop, charge what left the server, not what landed in the UI. Dashboards that ignore aborts teach you the happy path. Cost lives in the stop button.

A one-line check on a laptop is enough to start:

python envelope_tracer.py
# expect prompt_tokens to rise each turn faster than completion_tokens

If you want a pytest around the rule, assert the shape, not a vendor number.

def test_reprint_grows_faster_than_completion():
    env = Envelope(cap=50_000)
    transcript = [Turn("system", "x" * 400), Turn("user", "fix it")]
    assert env.charge_prompt(transcript, "ok")
    transcript += [Turn("assistant", "ok"), Turn("tool", "y" * 800)]
    assert env.charge_prompt(transcript, "still looking")
    first, second = env.log[0], env.log[1]
    assert second["prompt_tokens"] > first["prompt_tokens"]
    assert second["prompt_tokens"] > second["completion_tokens"]

That test is the whole ops note. If prompt tokens do not outrun completion tokens, you do not have a loop problem. You have a single-call problem, and your old per-job budget still works.

When free capacity is the wrong bet

Free model access is a rehearsal room. A free server is a quiet box for the tracer. Neither one changes the photocopy math. If the reprint curve is steep, moving the same loop onto a spare machine only moves the queue. The binder is still a binder.

You can run the envelope against MonkeyCode’s free model access and free server option when you need a real tokenizer in the loop instead of len(text) // 4. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use that lane to learn the turn where the reprint would have crossed the cap. Do not use it as a promise that an unbounded agent is suddenly cheap.

Free capacity is the wrong bet when the loop cannot abort. A job that must finish because a customer is staring at a spinner will sit behind whoever else is photocopying a repo. It is the wrong bet when tool fan-out is unbounded, when a search tool can paste nine files because the query was vague, and when the transcript carries secrets you should not park on a shared box. It is the wrong bet when you price success as “the agent returned,” because return values hide the cumulative prompt.

It is a reasonable bet when the task is throwaway measurement. A linter for your own tracer. A fixture that replays last week’s tool blobs. A comparison between “summarize the tool output” and “paste the file.” Those runs teach you whether to shrink the photocopies before you ever care which lane carries production traffic.

Shrink the photocopies on purpose. Summarize tool results before they re-enter the transcript. Replace file dumps with hashes plus the three failing lines. Cap search hits. Drop the system prompt’s novel if the tools already constrain behavior. Each of those cuts the reprint, which cuts every future turn, which is the only compounding discount this architecture has.

Do not confuse that with caching the same user prompt. Cache helps when the document is stable. Agent transcripts are not stable. They accrete. A cache key that includes the whole transcript is unique every turn, which is a polite way to say you still pay.

Who should not use this

Skip the envelope if you do not have a loop. A single completion with a known prompt size can use a boring max-tokens field and a boring bill. Skip it if you cannot abort; a cap you will override under pressure is decoration. Skip it if your tools return structured IDs and never prose. There is no photocopier in that design. You are already doing the right thing.

Also skip a shared free server if the transcript can contain customer code, credentials, or anything you would not paste into a ticket. Rehearsal is not a data-handling policy. Keep those jobs on iron you control, with the same abort, because the math does not care who owns the GPU.

The operational habit is small. Log prompt versus completion per turn. Cap the sum, not the step count. Abort in the client. Treat free capacity as a place to watch the curve, not a place to hide it. When the reprint is the bill, you stop arguing about models and start editing what you mail back to yourself.

If you try the tracer on a spare box, keep the envelope on. The number worth writing down is not tokens remaining. It is the turn index where the next photocopy would have failed the cap.

You Keep Paying for the Same Prompt

Quinn Li — Sat, 12 Sep 2026 18:31:53 +0000

You do not mainly pay for the next tool call. You pay to recite the conversation that justified it. That is the part most free-lane plans miss.

A chat loop looks like one job in your head. The API sees a growing document. Turn one is a short ask. Turn four is the ask, plus three tool results, plus your earlier reasoning, shipped again as input. The new thought is a rounding error. The echo is the bill.

Call it replay tax. You already spent tokens to produce a tool result. Then you spend again to remind the model that the result exists. If the model cannot see prior turns unless you resend them, every extra tool hop is also an extra copy of history. Free capacity does not cancel that copy. It only decides who waits while the copy is processed.

This is not an intelligence debate. It is operations. Public talk about agents keeps circling whether the loop is “real.” Cost ops should ask a colder question: what fraction of each request is new work, and what fraction is a photocopy of work you already did? Loop engineering fails in public when the loop never stops. It fails in private when each iteration inflates the prompt until your cheap lane is busy copying text you already paid to generate.

Picture a warehouse that restickers every crate before the forklift moves. The move looks cheap. The resticker line is not. Your context window is that line. The model is the forklift. Hiring another forklift does not shrink the labels. It just makes a longer line of machines waiting to read the same stickers.

So you split the meter. Stop logging “tokens per request” as one heroic number. Log two. Replay tokens are the prompt prefix that already existed before this turn’s new user or tool text. Fresh tokens are only the delta. If replay over fresh stays high, you are not thinking more. You are photocopying.

Here is a harness you can run locally. It does not call a vendor and it does not invent an invoice. It estimates tokens with a blunt chars/4 rule so you can see the shape of the bill before you spend real quota. Treat the counts as a ranking tool. Label them estimates until a real tokenizer sits in the same function.

"""Replay-tax harness. Estimates, does not bill."""
from dataclasses import dataclass, field
from typing import Literal

Kind = Literal["system", "user", "assistant", "tool"]

def est_tokens(text: str) -> int:
    # Rough, portable, wrong in the tails. Good enough to catch echo.
    return max(1, len(text) // 4)

@dataclass
class Turn:
    kind: Kind
    text: str

@dataclass
class LoopBill:
    turns: list[Turn] = field(default_factory=list)
    rows: list[dict] = field(default_factory=list)

    def add(self, kind: Kind, text: str) -> dict:
        fresh = est_tokens(text)
        replay = sum(est_tokens(t.text) for t in self.turns)
        total_in = replay + fresh
        ratio = replay / total_in if total_in else 0.0
        row = {
            "n": len(self.turns) + 1,
            "kind": kind,
            "fresh": fresh,
            "replay": replay,
            "input": total_in,
            "replay_ratio": round(ratio, 3),
        }
        self.turns.append(Turn(kind, text))
        self.rows.append(row)
        return row

    def should_stop(self, max_turns: int, max_input: int, max_ratio: float):
        if not self.rows:
            return None
        last = self.rows[-1]
        if last["n"] > max_turns:
            return "turn_cap"
        if last["input"] > max_input:
            return "input_cap"
        if last["n"] >= 3 and last["replay_ratio"] > max_ratio:
            return "echo_cap"
        return None


def demo_tool_loop() -> None:
    bill = LoopBill()
    bill.add("system", "You are a repo assistant. Prefer tools over guesses.")
    bill.add("user", "Why does checkout fail on order 1842?")
    tool_blob = "stack=trace " + ("frame " * 80)
    for i in range(1, 6):
        bill.add("assistant", f"Calling get_logs id={i}")
        bill.add("tool", tool_blob + f" id={i}")
        reason = bill.should_stop(max_turns=8, max_input=4000, max_ratio=0.85)
        print(bill.rows[-1], "stop=", reason)
        if reason:
            break

if __name__ == "__main__":
    demo_tool_loop()

Run it with python replay_tax.py. Watch fresh stay almost flat while replay stairs up. That stair is the argument against treating the loop as one cheap completion. You can swap est_tokens for a real tokenizer later. Do not wait for the perfect tokenizer to put a cap on the loop.

Caps belong in the same function that enqueues the next model call. A turn cap stops “just one more tool.” An input cap stops a log dump from eating the window. An echo cap is the one people skip. It fires when the conversation is mostly a recording of itself. Tune the 0.85 threshold if you must. Do not skip the measurement.

A tiny self-check keeps the rule honest. It is not a benchmark. It only proves the stair is visible.

def test_replay_outgrows_fresh():
    bill = LoopBill()
    bill.add("user", "short question")
    blob = "x" * 800
    bill.add("tool", blob)
    bill.add("tool", blob)
    last = bill.rows[-1]
    assert last["replay"] > last["fresh"]
    assert last["replay_ratio"] > 0.5
    assert bill.should_stop(8, 4000, 0.5) in {"echo_cap", "input_cap", None}

Wire the same counters around a real client the same way. Before you send, snapshot prompt size. After the tool returns, snapshot again. Log fresh, replay, wait_ms, and lane. The lane field is the cost-ops part. Free model access is a lane with a queue. A free server is a lane with noisy neighbors. Neither lane deletes replay tax. They only change how long you stand there while history is tokenized again.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option. That pairing is useful here as a lab, not as a production contract. Point the harness at a spare model, keep the loop on a free server, and learn whether your agent is thinking or reciting. Use the run to decide whether the next environment should even be free. If replay ratio climbs above your echo cap on turn three, a faster queue will not save you. You need a smaller transcript.

When is free capacity the wrong bet for this failure mode? When the job is a loop whose prompt is dominated by prior turns, and a human is waiting on the answer. You are not buying a better thought. You are buying another copy operation behind other copy operations. The wait is real. The tokens are mostly echo. Putting that work on a free lane trains you to ignore both. Deadline work has the same smell, but the mechanism is different: here the clock dies because the prompt is a novel you insist on rereading aloud.

Compress before you resend. Tool output is the usual villain. A fat stack trace you include “for context” becomes a fatter prompt two turns later. Summarize the tool result into the fact you actually need. Keep the raw dump in your own store. The model does not need the dump twice. It needs the conclusion and a pointer.

Drop a clip in front of bill.add("tool", ...) so the warehouse stops accepting unlabeled crates.

def clip_tool(raw: str, keep: int = 600) -> str:
    raw = raw.strip()
    if len(raw) <= keep:
        return raw
    head, tail = raw[: keep // 2], raw[-keep // 2 :]
    return f"{head}\n...\n{tail}\n[clipped {len(raw) - keep} chars]"

Clipping is not insight. It is inventory control. You are deciding what re-enters the prompt. If you cannot state the fact in two sentences, you are not ready to pay replay tax on the full blob. The same rule applies to your own assistant messages. Long chain-of-thought pasted back in as “memory” is just echo with better manners.

A second control is checkpointing the goal, not the chat. After two tool turns, write a standing brief: question, facts so far, next action, stop condition. Start the next model call from that brief plus the new tool result. You throw away the theater. You keep the state. This is the opposite of “let the agent see everything.” Seeing everything is how echo becomes the product.

BRIEF = """Question: {q}
Facts: {facts}
Next: {nxt}
Stop if: {stop}
"""

def next_prompt(q, facts, nxt, stop, latest_tool: str) -> str:
    body = BRIEF.format(
        q=q,
        facts="; ".join(facts[-6:]),
        nxt=nxt,
        stop=stop,
    )
    return body + "\nLatest tool:\n" + clip_tool(latest_tool)

That function is the cost control. The model still gets a loop. You no longer resend the novel. If a later turn needs a raw log, fetch it from your store as a new tool call, clip it, and forget it again. Do not keep a souvenir.

Debug it like an incident, not like a prompt-craft session. First print the last row of LoopBill on every turn. If replay_ratio crosses your cap, dump the turn kinds, not the full text. You want to see tool, tool, tool, not another stack trace in your pager. Second, time the send. If wait_ms grows while fresh does not, you are queued on echo. Third, only then change lanes. Moving an obese transcript onto a “better” model is how teams spend money to copy faster.

Limitations are sharp. chars/4 will mis-rank CJK text, minified code, and tokenizer-specific specials. Echo ratio is a heuristic; a retrieval turn can look fresh while still being junk. Caps can cut off a legitimate long investigation. Tokenizer-free estimates will not match a vendor invoice. If you ship customer-facing agents with legal or safety constraints, do not use a free lane as the only backstop. Put a hard server-side turn limit in front of any model, including a paid one.

Who should not use this approach: anyone who needs a hard latency SLO on the next token, anyone whose tool output is the artifact itself (you cannot clip a contract and call it done), and anyone who has not yet proven the loop terminates. If the product is “keep calling tools until the user is happy,” you do not have a budget. You have a vibe. Free model access will hide the vibe until the queue does not.

Do not confuse this note with a claim about model quality. A stronger model can still drown in its own minutes. A weaker model on a short brief can finish. Your job is to stop paying for the minutes. Measure replay. Cap the echo. Keep free capacity for the experiments that still fit in a small prompt. When the transcript is the product, pick a lane that prices waiting honestly, or stop looping.

If you need a quiet bench to watch the ratio move, MonkeyCode’s free model access and free server option are enough to run the harness against a real loop. Treat the session as measurement, then take the echo cap with you even if you never use that lane again.