Give Free Capacity a Deadline

Give free capacity a deadline, or it will spend yours. A token counter cannot tell you that the job sat in a queue until the deploy window closed. You need a second meter, and you need that meter to be allowed to kill the work.

Tokens are a quantity. A release is a time. Those units do not convert. Teams that only watch spend dashboards learn this on a Thursday afternoon, when the cheap path is still “running” and the humans have already lost the day.

Think of a grocery express lane that is free and unstaffed. You do not pay at the register. You pay with the melting ice cream. Interactive coding, review replies, and anything with a person waiting on the other side of the cursor is ice cream. Overnight evals can wait in that lane. The mistake is mixing them because both happen to call a model.

You already know the token clock. Prompt tokens in, completion tokens out, maybe a retry multiplier if you are honest. The calendar clock is cruder and more expensive: enqueue time, first-byte time, done time, and the wall time you promised a human. Free model access and a free server option move cost off the token clock. They do not stop the calendar.

If you are exploring that pair, treat it as opportunistic capacity, not as an SLA. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one place that currently offers free model access and a free server option. The wrapper below does not depend on it. Point the same timer at any endpoint you already run.

Here is the operational claim, stated plainly. You may use spare capacity when the job can die without breaking a promise. You may not use it when a person, a deploy, or a customer ticket is the clock. Hope is not a queue policy. A deadline is.

The proposed control loop is small. Stamp the job when you intend to start, not when the model first speaks. Separate wait from generation. If wait exceeds the budget, abort before you generate a novel. If generation exceeds the budget, abort even if the answer is mid-sentence. Log both clocks in one JSON line so tomorrow-you can argue from a file instead of a feeling.

The script below is an unexecuted example. It talks to whatever URL you put in INFER_URL. Dry-run mode sleeps so you can rehearse the kill switch without credentials or a live model.

#!/usr/bin/env python3
"""wallclock_budget.py — proposed timer for any HTTP inference call."""
import argparse, json, os, sys, time, urllib.error, urllib.request

def emit(event, **fields):
    row = {"event": event, "ts": time.time(), **fields}
    sys.stdout.write(json.dumps(row, separators=(',', ':')) + "\n")
    sys.stdout.flush()

def post(url, body, timeout):
    req = urllib.request.Request(
        url,
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return resp.status, json.loads(resp.read().decode() or "{}")

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--wait-budget-s", type=float, default=20)
    p.add_argument("--total-budget-s", type=float, default=60)
    p.add_argument("--dry-run", action="store_true")
    p.add_argument("--simulate-wait-s", type=float, default=0)
    p.add_argument("--prompt", default="Summarize the failing test in one paragraph.")
    args = p.parse_args()
    url = os.environ.get("INFER_URL", "http://127.0.0.1:8080/v1/complete")
    t0 = time.monotonic()
    emit("enqueued", wait_budget_s=args.wait_budget_s, total_budget_s=args.total_budget_s)

    if args.simulate_wait_s:
        time.sleep(args.simulate_wait_s)
    waited = time.monotonic() - t0
    emit("first_attempt", wait_ms=int(waited * 1000))
    if waited > args.wait_budget_s:
        emit("aborted", reason="wait_budget", wait_ms=int(waited * 1000))
        return 2

    remaining = max(0.1, args.total_budget_s - waited)
    try:
        if args.dry_run:
            time.sleep(min(0.2, remaining))
            status, payload = 200, {"dry_run": True, "text": ""}
        else:
            status, payload = post(url, {"prompt": args.prompt}, timeout=remaining)
    except urllib.error.URLError as exc:
        emit("aborted", reason="transport", error=str(exc), wait_ms=int(waited * 1000))
        return 3

    total = time.monotonic() - t0
    gen_ms = int((total - waited) * 1000)
    if total > args.total_budget_s:
        emit("aborted", reason="total_budget", wait_ms=int(waited * 1000),
             gen_ms=gen_ms, http_status=status)
        return 2
    emit("completed", wait_ms=int(waited * 1000), gen_ms=gen_ms,
         total_ms=int(total * 1000), http_status=status,
         chars=len(json.dumps(payload)))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Rehearse the abort on your laptop before you aim it at a shared box.

chmod +x wallclock_budget.py
python3 wallclock_budget.py --dry-run --simulate-wait-s 25 --wait-budget-s 20
echo exit:$?
python3 wallclock_budget.py --dry-run --simulate-wait-s 2 --wait-budget-s 20 --total-budget-s 60

The first command should exit 2 and print aborted with wait_budget. That is the point. You wanted the failure in twenty seconds, not a silent wait that turns into an unreviewed diff at 6 p.m. The second command should complete. Same prompt, same “free” path, different relationship to the calendar.

When you have a real endpoint, keep the logs. One line per attempt is enough to see whether you are paying in queue or in generation.

export INFER_URL='http://127.0.0.1:8080/v1/complete'
python3 wallclock_budget.py --wait-budget-s 15 --total-budget-s 45 \
  --prompt 'List the three failing assertions and nothing else.' \
  | tee -a infer-clock.jsonl

python3 - <<'PY'
import json
waits, gens, aborts = [], [], 0
for line in open("infer-clock.jsonl"):
    row = json.loads(line)
    if row.get("event") == "aborted":
        aborts += 1
    if "wait_ms" in row and row.get("event") in {"completed", "aborted"}:
        waits.append(row["wait_ms"])
    if row.get("event") == "completed":
        gens.append(row.get("gen_ms", 0))
print("aborts", aborts)
print("p50_wait_ms", sorted(waits)[len(waits)//2] if waits else None)
print("p50_gen_ms", sorted(gens)[len(gens)//2] if gens else None)
PY

Read that file like an invoice with two columns. If wait dominates, you did not get a cheap model. You got a slow ticket counter. If generation dominates and you still miss the window, your prompt is too fat for the remaining minutes. Either way, the token dashboard was never going to show it.

A curl one-liner is enough when you do not want Python in the loop. It is worse at separating wait from generate, and that is the lesson. Wall time without a split is how “the model was slow” becomes an unexamined story.

START=$(date +%s)
curl -sS -m 45 -H 'Content-Type: application/json' \
  -d '{"prompt":"Return only the stack frame that raised."}' \
  "$INFER_URL" >/tmp/infer.out
END=$(date +%s)
echo elapsed_s:$((END-START))

Use the split when the job can wait. Use the blunt curl timeout when you already know the human is in the loop. Do not put an editor-backed request on a path you would not give a p99 to.

The decision is not ideological. It is a shape. A nightly corpus scan with hours of slack can ride spare capacity, including a free server, because a kill at 2 a.m. costs you a rerun, not a meeting. A “fix this test before standup” request cannot. A customer-facing complete-me box cannot. A regulated workload that needs tenancy and an audit trail cannot. Same model family, different clock.

Job shape	Slack on the calendar	Kill switch	Spare / free path
Overnight eval sweep	Hours	Retry tomorrow	Fit
Prompt regression, no human waiting	Tens of minutes	Abort and narrow the batch	Fit if wait stays inside slack
PR comment, pairing, incident reply	Seconds to a few minutes	Fail closed, switch off spare	Poor fit
User-facing completion	Hard p99	Do not abort into a blank UI	Wrong tool

Notice what the table does not say. It does not say free is fake. It does not say paid is virtuous. It says the calendar is the constraint that tokens cannot express. When slack is large, opportunistic capacity is a rational bet. When slack is a standup, you are betting the meeting.

Limitations are not fine print. This wrapper does not see the provider’s internal queue, so wait_ms starts when your process starts, not when their scheduler notices you. Abort may still consume work the remote side already began; a timeout is not a refund. Clocks drift across machines, so run the timer next to the client, not on a dashboard in another region. Dry-run sleeps are not a load test. A free tier can change shape without updating your script. None of this replaces a reserved endpoint for production traffic.

Skip this approach if you ship interactive inference to other people. Skip it if a missed completion is a safety event. Skip it if your org cannot tolerate preemption, noisy neighbors, or a job that vanishes mid-token. Skip it if you were about to use spare capacity to “move faster” on a deadline you already quoted. Speed you cannot timestamp is not speed.

AI-assisted coding makes the mix-up easier, because the editor feels interactive while the remote job is a batch queue in costume. Calling that engineering does not make the ice cream freeze. Measure wait. Cap the total. Keep deadline work off the unstaffed lane. If you already have a free-capacity path, wrap one real job in the timer this week and read wait_ms before you argue about token price.