Book the Partial Before You Retry

A free model call that you abandon is not free. You already spent the queue, and you already spent the tokens the server emitted before you cut the socket. Write that partial off before you retry. If you do not, you are running a loop that looks thrifty and bills you in minutes.

You hit this when a shared free server is slow and your client is optimistic. The first attempt streams a couple hundred tokens, then the deadline you promised a person slips, so you cancel and send the same prompt again. The second attempt waits behind work you did not actually pull out of the pool by closing the socket. A zero price on the token meter does not rewind that queue.

It only hides the wait inside a dashboard that still says free. The retry question is live in developer discussions again. Treat it as a ledger problem, not as a mood your HTTP client is in.

Picture a print shop that takes night jobs at no charge. You submit a chapter, they set type for three pages, then you cancel because the pickup window moved. Those pages stay on the bench. Submitting the chapter again adds a second job behind the first.

What you book

Keep three numbers on the job, and update them after every attempt. spent_s is queue plus generation you already waited through. spent_tokens is what came back, including a partial you will not ship. ceiling_tokens is a cap you chose for this job, not a vendor quota copied out of a post.

The cap exists so a free lane cannot mean an unbounded loop inside your own process. A retry on that lane is allowed only when the next attempt can finish inside the remaining deadline, and only when spent tokens plus a fresh worst-case estimate still fit inside the ceiling. Otherwise you cut over to a paid lane for one clean attempt, or you stop and return what you have.

Stopping is a result. Another hopeful enqueue is not. Pass an empty attempt list for the first call, so the same function admits or refuses before you have a partial and you are not maintaining two policies.

The figures in the fixtures are placeholders you replace with timings you took this week. Nothing here is a measurement of a public pool, a price list, or a limit that will still be true next Monday. If a number is older than your last sample, it is already the wrong input.

A ledger you can run

Save the script as book_partial.py. It is a local decision helper, not a client for any vendor, and it will not open a socket. You feed it history. It returns an action.

#!/usr/bin/env python3
"""Write off partial free-lane attempts before you allow another retry."""

from __future__ import annotations

import json
import sys
from dataclasses import dataclass


@dataclass(frozen=True)
class Attempt:
    queue_s: float
    gen_s: float
    emitted_tokens: int
    shipped: bool


@dataclass(frozen=True)
class Policy:
    deadline_s: float
    ceiling_tokens: int
    next_queue_s: float
    next_gen_s: float
    next_est_tokens: int
    paid_queue_s: float
    paid_usd_per_1k: float


def book(attempts: list[Attempt], policy: Policy) -> dict:
    spent_s = sum(a.queue_s + a.gen_s for a in attempts)
    spent_tokens = sum(a.emitted_tokens for a in attempts)
    remaining_s = policy.deadline_s - spent_s
    next_free_s = policy.next_queue_s + policy.next_gen_s
    next_paid_s = policy.paid_queue_s + policy.next_gen_s
    projected = spent_tokens + policy.next_est_tokens

    if any(a.shipped for a in attempts):
        action, why = "stop", "a previous attempt already shipped"
    elif projected > policy.ceiling_tokens:
        action, why = "stop", "another try would cross your token ceiling"
    elif next_free_s <= remaining_s:
        action, why = "retry_free", "remaining clock covers one more free attempt"
    elif next_paid_s <= remaining_s:
        action, why = "cut_over", "free retry does not fit; one paid attempt does"
    else:
        action, why = "stop", "no lane finishes inside the time you have left"

    sunk_usd = (spent_tokens / 1000.0) * policy.paid_usd_per_1k
    return {
        "action": action,
        "why": why,
        "spent_s": round(spent_s, 2),
        "remaining_s": round(remaining_s, 2),
        "spent_tokens": spent_tokens,
        "projected_tokens": projected,
        "sunk_usd_if_those_tokens_were_paid": round(sunk_usd, 4),
    }


def main() -> int:
    raw = json.load(sys.stdin)
    attempts = [Attempt(**item) for item in raw["attempts"]]
    policy = Policy(**raw["policy"])
    json.dump(book(attempts, policy), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The first fixture is a night job with a wide clock. One partial came back, nothing has shipped, and you still have room under a ceiling you set yourself.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

You should see retry_free. Spent time is 44 seconds, the remaining window is wide, and 220 plus a guessed 800 stays under 4000. The dollar field is a shadow price so the partial stays visible on the record. It is not an invoice, and the rate is a stand-in you must replace.

If you do not know a current paid rate, set paid_usd_per_1k to 0 and decide on time and tokens alone. A stale price copied from an old thread is worse than a zero, because it looks like evidence.

Now the interactive case. Same partial, a 60 second deadline, and the same next guess. The night job and the inline job are not the same bet, even when they share a prompt.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 60,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

Remaining time is 16 seconds. The next free attempt wants 47. One paid attempt wants 14, which still fits, so the action is cut_over. The free server did not become a bad product in that branch; the remaining clock stopped matching the queue you would sit in again.

If you lower the ceiling under the projection, the same function stops instead of retrying, even when the deadline is wide. Run the third fixture and read the action before you argue with it.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 500,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

Projected tokens are 1020 against a ceiling of 500, so you stop. A long deadline does not authorize an open token loop just because the lane advertises free model access. Your ceiling is how you keep that advertisement from becoming a retry policy. Count tokens the same way on every attempt, or the stop branch fires late and you will blame the wrong line.

Pin the branches

Wire a tiny assertion so the three branches cannot drift while you edit the formula. This is a local test. It does not touch a network, and it fails if you reorder the checks without updating the expected actions.

python3 - <<'PY'
import importlib.util

spec = importlib.util.spec_from_file_location("book_partial", "book_partial.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)

def run(deadline, ceiling, emitted=220):
    attempts = [mod.Attempt(35, 9, emitted, False)]
    policy = mod.Policy(deadline, ceiling, 35, 12, 800, 2, 0.002)
    return mod.book(attempts, policy)["action"]

assert run(600, 4000) == "retry_free"
assert run(60, 4000) == "cut_over"
assert run(600, 500) == "stop"
assert run(60, 500) == "stop"
print("ok")
PY

Run that from the directory where you saved the file, then read the last assert with the cut-over case beside it. A tight deadline and a blown ceiling both stop, and stop wins over cut-over because the ceiling check runs first. If you would rather pay once the free path has already burned a partial, swap those branches and re-run the test until it matches the policy you actually want. A client that retries on its own will not honor either order, so turn that off or the ledger is a story you tell after the loop has already run.

Sample the queue you will sit in

Before you trust next_queue_s, take a sample in the hour you actually run. Send a short fixed prompt on the free lane, record time to first token as queue, and record the rest as generation. Do that three times, keep the worst queue, and discard the probe text. If the probe fails, the lane is not eligible this hour.

Do not retry the probe until it feels fine. That loop is the one the ledger is meant to stop. The command below is a stand-in timer, not a benchmark and not a vendor call. Replace the sleep with your own client, then put the worst sample into the policy.

python3 - <<'PY'
import time

start = time.perf_counter()
# Stand-in only. Replace this sleep with one short probe call.
time.sleep(0.05)
queue_s = time.perf_counter() - start
print(f"sample_queue_s={queue_s:.3f}")
PY

Where a free lane still belongs

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode belongs in this workflow only as a spill lane: free model access and a free server option, as supplied for this note, for jobs the ledger has already marked retry_free. This page does not name models, token grants, hardware, or how long either option lasts, because those facts move and a post should not freeze them. If you still want that spill lane, read the live limits, copy the ones you will depend on into ceiling_tokens and next_queue_s, and run the assertion again when the sample changes.

Free capacity is the wrong bet when a person is blocked on the answer, when the attempt you would abandon is not idempotent, or when you cannot sample queue time and would be typing a wish into next_queue_s. It is also the wrong bet when a retry can double-apply a side effect, since booking tokens will not undo a second write. Stamp your own idempotency key on the job before the first call. Treat shipped: true as final, so a late free response cannot overwrite a paid answer you already returned.

What this leaves out

The function trusts your samples. One quiet afternoon is not a week, so if the queue swings, keep the worst sample from the hour you run and put that number in next_queue_s. It also treats every emitted token as spent, which is the right bias for a ceiling and the wrong bias for a quality score. A partial you can resume is not the same shape as a partial you must throw away.

If your client supports continuation, add only the missing gap to the projection, then re-run the assertion so the new bias is the one you tested. Do not use this for payments, access control, or any path where stop is not an acceptable outcome and you have no contracted fallback. Do not use it as a capacity forecast for a team. It books one job's history, and it will not tell you whether a free server can hold everyone at the top of the hour.

You will still be tempted to skip the write-off, because the dashboard said the call was free and the partial looked small. Book it anyway. The next retry is where a free lane turns into a queue you cannot see the end of.