Write the Prefix Down Before the Slot Dies

A preempted stream is not a free retry. If the lane already emitted tokens, those tokens are spent, and starting the prompt over buys a prefix you already hold.

You feel the urge anyway. The socket drops, the free slot vanishes, and the editor still shows an unfinished function. The cheap instinct is to send the same messages again and hope the next slot is kinder. That instinct turns a short outage into a second bill for the same answer.

Think of a courier who already walked the parcel to your door, then got pulled off the route. You do not hire a second courier to repeat the walk from the warehouse. You keep what arrived and pay only for the last flight of stairs.

A model stream has that shape, with one wrinkle. Input context may have to be shown again, because a provider often will not hand you a receipt for cached prompt tokens. Output you already received must not be generated again. Those are different costs, and mixing them up is how the bill grows while the answer stays the same.

This note is a small ops habit for that split. It is a proposal, not a measured result, and the sample is unexecuted example code. It does not assume a model name, a quota, a machine size, or a promise that a free lane will exist next week.

Before the call, write a job row. Give it an id, a hash of the prompt, a token budget, and a cursor at zero. The hash is how you refuse to resume against a prompt you have since edited. If the hash does not match, the cursor belongs to a different question, and stitching it on would answer the wrong ask.

While tokens arrive, flush the cursor on a short interval, not only at the end. A flush at the end is a diary of success. A flush every few dozen tokens is a diary of the failure you actually have.

Store the assistant text, the count you treat as spent, and a monotonic revision. If the process dies between flushes, you lose only the gap. That gap is the acceptable loss. The prefix before it is not.

On the way back in, do not rebuild the original request and ask for a fresh completion. Build a continuation, and cap output at budget minus spent. If the API allows an assistant prefill, put the stored prefix there and ask it to continue without repeating itself.

If the API will not accept a prefill, send a short instruction plus the unfinished tail, not the entire original thread plus a demand for a new essay. If the remainder is below a floor you choose, abandon. A continuation that may spend the original budget again is just a replay wearing a new function name.

Here is the gate in plain Python. Save it as budget_cursor.py. Treat every number as a knob, not a measurement. The annotations assume a current Python 3.

import hashlib, json
from pathlib import Path

ROOT = Path(".jobstate")

def prompt_hash(messages: list[dict]) -> str:
    blob = json.dumps(messages, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()

def open_job(job_id: str, messages: list[dict], budget_tokens: int) -> dict:
    ROOT.mkdir(exist_ok=True)
    path = ROOT / f"{job_id}.json"
    if path.exists():
        job = json.loads(path.read_text())
        if job["prompt_hash"] != prompt_hash(messages):
            raise SystemExit("prompt changed; refuse stale cursor")
        return job
    job = {
        "job_id": job_id,
        "prompt_hash": prompt_hash(messages),
        "budget_tokens": budget_tokens,
        "spent_tokens": 0,
        "prefix": "",
        "revision": 0,
        "state": "open",
    }
    path.write_text(json.dumps(job))
    return job

def flush(job: dict, piece: str, piece_tokens: int) -> dict:
    job["prefix"] += piece
    job["spent_tokens"] += piece_tokens
    job["revision"] += 1
    if job["spent_tokens"] > job["budget_tokens"]:
        job["state"] = "over_budget"
    path = ROOT / f"{job['job_id']}.json"
    tmp = path.with_suffix(".json.tmp")
    tmp.write_text(json.dumps(job))
    tmp.replace(path)
    return job

def continuation(job: dict, floor: int = 32) -> dict | None:
    remain = job["budget_tokens"] - job["spent_tokens"]
    if job["state"] != "open" or remain < floor or not job["prefix"]:
        return None
    return {
        "assistant_prefill": job["prefix"],
        "max_output_tokens": remain,
        "instruction": "Continue from the prefill. Do not repeat it.",
    }

The temp-file rename avoids a half-written JSON line if the process dies mid-flush on the same filesystem. It is not a distributed lock. Two workers sharing one job id will still clobber the cursor, so give each in-flight call its own id.

You can exercise the refusal path with no network and no key. The 12 below is a stand-in, not a tokenizer result. In a real call you store the usage figure for that chunk, not a guess from string length.

mkdir -p .jobstate
python - <<'PY'
from budget_cursor import open_job, flush, continuation
msgs = [{"role": "user", "content": "draft a migration note"}]
job = open_job("mig-14", msgs, budget_tokens=400)
flush(job, "Step 1. Freeze writes.\n", 12)
print(continuation(job))
print("remain", job["budget_tokens"] - job["spent_tokens"])
PY

If continuation returns None, you stop. You do not send the original call once more just in case. None means the prefix already consumed the useful budget, or there is no prefix worth resuming. A full replay would ignore the ending you just computed.

Watch the file if you want to see the cursor move under a fake stream. This loop is still local. It proves the flush, not a provider.

python - <<'PY'
import json
from budget_cursor import open_job, flush
job = open_job("mig-14", [{"role": "user", "content": "draft a migration note"}], 400)
for piece, n in [("Step 2. Copy rows.\n", 10), ("Step 3. Verify counts.\n", 11)]:
    flush(job, piece, n)
print(json.dumps(job, indent=2))
PY

You have three exits, and you should pick one before the call, not during the panic. Abandon when the remainder is under the floor or the prefix is empty. Resume when the prefix is real and the remainder still fits the budget.

Spill to a lane you are willing to wait on only when the job cannot be dropped and the free slot has already failed. Spilling is not a loop. One spill, then a human, or a queue that can actually reserve the work. The reservation problem is older than this cursor, and this note does not solve it.

A free lane is a decent stage for that rehearsal, and a bad home for work you cannot drop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The useful fit is narrow: MonkeyCode's free model access and a free server option can host a broken-stream drill, after you read the live limits in the product you have, and only for jobs you can abandon. This note does not record a quota, a card, a region, or a duration, because those figures move and no primary source for them was attached.

Free capacity is the wrong bet when the remainder is not optional. A migration note that can be rewritten later may die at a cursor. A payment capture, a mailbox send, an infrastructure apply, or a customer-visible webhook may not. Those need an idempotency key, and the side effect should run after the text is finished, not inside the stream. If a partial answer is worse than silence, do not emit it onto a lane that can vanish mid-sentence.

There is a quieter wrong bet than preemption: input replay. If the provider bills the prompt again on every continuation, a huge system prompt will swallow the savings from not regenerating the prefix. Log input tokens and output tokens separately on the resume call. If resumed input is nearly the original prompt plus the stored prefix, you moved the cost, you did not remove it.

Shrink the resumed context to the tail you still need, or stop using continuation for that prompt shape. The cursor still earns its keep when it blocks a second full completion. It will not make a bloated prompt cheap, and it will not refund tokens a vendor has already metered.

Skip this habit if you cannot persist the prefix outside the process. A cursor that lives only in memory dies with the preempt and leaves the replay you were trying to avoid. Skip it when the client hides token counts and you would be guessing piece_tokens. A guessed cursor is a lie in the ledger.

Skip it when the attempt must be byte-identical to the first sample. Continuation is a new draw, not a restored sampler, unless the provider documents otherwise. Do not claim determinism you did not buy. And skip it for any call that mutates the world before the stream ends, because a resume would be a second chance to mutate it.

The habit is small enough to keep. Open a job, flush while text arrives, and on death ask only for the remainder or ask for nothing. A free slot can host the drill. It cannot refund a prefix you already spent, and it cannot be the commitment under work you are unwilling to lose.