You should not put a customer-facing job on free model access or a free server. The invoice looks kind. The calendar does not. Free capacity behaves like a spot instance: it is real, it is useful, and it can vanish or stall without warning the moment you needed a percentile, not an average.
That is the whole note. The rest is how you stop mixing the two lanes.
Cloud engineers already learned this with interruptible VMs. You compile on them. You crawl on them. You do not put checkout on them. Language-model work forgot the lesson because the unit of waste is quieter. A preempted VM leaves a hole in a fleet dashboard. A stalled generation leaves a half-built answer, a warm context you cannot reuse, and a human still waiting in the same ticket.
Name the work before the first token. Speculative jobs are searches, prompt comparisons, throwaway extractions, overnight summaries nobody will page you about. Interruption is annoying. It is not a business event. Committed jobs are different. A user is blocked, a pipeline gate is red, or a room is waiting on a number. From that side of the glass, retries are still the same job. The scarce resource is not tokens. It is the remaining minutes before the room goes cold.
You already know the failure mode. The job “looks small,” so you park it on the free lane. The queue is polite for twenty minutes. Then the worker thrashes, the context is rebuilt from scratch, and you save a few cents by missing a release train. Cheap on the bill. Dear on the week. Free capacity is the wrong bet the instant someone else’s clock is attached to the result.
The hidden line item is partial work. Free pools are not obligated to finish. You pay in tokens that produced no success, in queue time that aged the request, and in the cognitive tax of checking whether the job is slow or dead. None of that prints on a vendor invoice. It still hits the same budget: attention. A lane that dies at 90% complete is not a discount. It is an unbilled retry factory.
So you need a router, not a vibe. Classify the job before you spend context. Record waste when the free lane abandons you. Promote or fail fast. Do not “see how it goes” on a path that has a deadline. The snippet below is a proposal you can run locally. It is not a vendor SLO, and it does not pretend to know how any free pool schedules work.
from dataclasses import dataclass
from enum import Enum
class Lane(str, Enum):
FREE = "free"
GUARANTEED = "guaranteed"
class JobClass(str, Enum):
SPECULATIVE = "speculative"
COMMITTED = "committed"
@dataclass
class Job:
name: str
job_class: JobClass
token_budget: int
wall_deadline_s: float
customer_blocked: bool
@dataclass
class Attempt:
lane: Lane
tokens_used: int
elapsed_s: float
completed: bool
preempted: bool
class WasteLedger:
def __init__(self) -> None:
self.wasted_tokens = 0
self.completed_tokens = 0
self.preemptions = 0
def record(self, attempt: Attempt) -> None:
if attempt.completed:
self.completed_tokens += attempt.tokens_used
return
self.wasted_tokens += attempt.tokens_used
if attempt.preempted:
self.preemptions += 1
def waste_ratio(self) -> float:
total = self.wasted_tokens + self.completed_tokens
return 0.0 if total == 0 else self.wasted_tokens / total
def route(job: Job) -> Lane:
# Committed work never starts on spot-like capacity.
if job.job_class == JobClass.COMMITTED or job.customer_blocked:
return Lane.GUARANTEED
if job.wall_deadline_s < 30:
return Lane.GUARANTEED
return Lane.FREE
def should_stop_using_free(ledger: WasteLedger) -> bool:
# Gray jobs only. If free keeps dying, stop arguing with it.
return ledger.waste_ratio() > 0.35 and ledger.preemptions >= 2
Read the policy the way you would read a spot-fleet rule. Speculative jobs may sit on free. Committed jobs never start there. A short wall clock is a committed signal even if you labeled the ticket “research.” If you insist on a hybrid for a gray job, the waste ledger is the adult in the room. A free attempt that burns 8k tokens and then disappears did not save you 8k tokens. It borrowed them from the next guaranteed run.
Wire a heartbeat if your runner can. A free server that stops heartbeating is a preemption, not a thoughtful model. Promote the job or fail it. Do not wait for the HTTP client timeout to teach you the same fact ten minutes later. You can exercise the classifier without calling any network at all:
python - <<'PY'
from cost_router import Job, JobClass, Lane, route, WasteLedger, Attempt, should_stop_using_free
release = Job("release-notes", JobClass.COMMITTED, 4000, 120, True)
explore = Job("prompt-bakeoff", JobClass.SPECULATIVE, 12000, 1800, False)
assert route(release) is Lane.GUARANTEED
assert route(explore) is Lane.FREE
ledger = WasteLedger()
ledger.record(Attempt(Lane.FREE, 8100, 94, False, True))
ledger.record(Attempt(Lane.FREE, 6400, 71, False, True))
assert should_stop_using_free(ledger) is True
print("waste_ratio", round(ledger.waste_ratio(), 3))
print("committed_lane", route(release).value)
PY
Keep a one-line log you can grep when a “cheap” afternoon turns expensive. Tokens without a success are not a research souvenir. They are a failed attempt, same as a killed container.
printf '%s\n' \
'ts=2026-09-20T18:04:11Z job=prompt-bakeoff lane=free tokens=8100 completed=0 preempted=1' \
'ts=2026-09-20T18:06:02Z job=release-notes lane=guaranteed tokens=1900 completed=1 preempted=0'
Add a tiny test so the rule cannot drift into “just this once.” Once is how committed traffic leaks onto spot.
from cost_router import Job, JobClass, Lane, route
def test_blocked_user_never_starts_on_free():
job = Job("support-reply", JobClass.SPECULATIVE, 800, 600, True)
assert route(job) is Lane.GUARANTEED
def test_true_speculation_may_use_free():
job = Job("corpus-skim", JobClass.SPECULATIVE, 20000, 7200, False)
assert route(job) is Lane.FREE
Run pytest -q in the same directory. If that test ever fails because someone “needed the free lane for a demo,” you have found the incident before production finds it. Demos with an audience are committed work. Treat them that way.
Where does a free product actually belong in this picture? On the speculative lane, and only there. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is a reasonable place to park bakeoffs, schema guesses, and overnight skims—the jobs you can afford to lose. It is the wrong place to park a customer-blocked reply or a release-train gate. If you try it, keep the router above in front of the client so curiosity and commitments do not share a queue.
The numbers will lie if you let them. Waste ratio ignores human time. A free job that completes after forty minutes can still be a miss against a ten-minute SLA, and a guaranteed job that spends more tokens can still be cheaper in calendar cost. Do not turn the ledger into a beauty contest between unit prices. Compare finished work against the clock that actually matters.
This policy is the wrong tool if you have no customer, no gate, and no clock. A weekend toy can live entirely on free capacity. A research corpus that may be thrown away tomorrow can too. Do not build a two-lane router for a script you run twice. The overhead of classification only pays for itself when mixed traffic would otherwise hide a stall inside a “savings” story.
Limitations are blunt. You cannot observe another operator’s preemption policy from your laptop. Heartbeats can lie. Token counts can lag. A job labeled speculative on Monday becomes committed on Tuesday when a stakeholder starts waiting. Re-classify at enqueue time, not at idea time. And never use free-lane success on a quiet afternoon as evidence it will hold a percentile on a busy one. Spot markets look generous until they do not.
If you take nothing else: invoice cost and delivery cost are different currencies. Convert before you route. Park speculation on free. Buy the path that ships.
Top comments (1)
"A free lane that dies at 90% complete is not a discount. It is an unbilled retry factory." — this is the framing that makes the whole thing click. The invoice says you saved money. The calendar says you missed the window.
The re-classification rule at the end is what most implementations miss: a job labeled speculative on Monday becomes committed on Tuesday when a stakeholder starts waiting. "Re-classify at enqueue time, not at idea time" is the rule that actually prevents the leak, not the initial routing logic.
The waste ledger as a first-class concept — tracking preemptions separately from failures, computing a waste_ratio — is the right instrumentation. Most teams measure token cost. Almost nobody measures tokens-without-a-success, which is the number that actually predicts whether the spot lane is costing them more than guaranteed would.
The test pinning "support-reply with customer_blocked=True always routes GUARANTEED" is the key invariant. If that test ever breaks because "someone needed the free lane for a demo" — and demos with an audience are committed work — you've found the incident before production finds it.
At Black Label we're routing different agent jobs across capacity tiers and this maps directly to decisions we're making now.