You do not earn a model call by noticing that a free lane looks idle. You earn it when a fresh cost envelope still fits the lane, the deadline, and the meter you are willing to owe. Idle capacity is a rumor. An envelope is a constraint you can fail in a unit test before anyone spends a token.
A coat-check ticket is the right picture. It names the coat, the window, and the door. It does not mean the rack still has a hook when you wander back from the sidewalk. A free lane can be a real place to hang a small probe, and it can also be full, slow, or simply the wrong door for work you already promised a person.
This note is the bouncer in front of that door. You stamp a cost envelope, admit the job to a probe lane or a committed lane, and throw the ticket away when the attempt ends. A retry does not walk in on the old stub.
Reserving a place in line is not the same job as pricing the socket. You can hold a slot and still open an unbounded completion. The queue then looks healthy while a transcript you did not mean to authorize keeps the meter warm. The envelope is the missing half. It is the budget around the prompt, not the prompt, and not the reservation.
What has to be on the ticket
The envelope names an input-token cap, an output-token cap, a wall-clock timeout, an attempt number, a lane, and whether a human deadline is attached. Miss any of those and you are guessing. Guessing is how a five-minute experiment becomes an open tab.
Keep the token numbers local and dull. Count input tokens with a tokenizer you already trust, or with a character heuristic you have checked against your own logs. Label the heuristic as a heuristic. A character count is not a vendor invoice. You are refusing to cross a ceiling, not predicting that the answer will be good.
A probe envelope stays narrow on purpose. Short output cap. One attempt. No inherited retry. No customer deadline. A committed envelope may be wider, but it has to name a meter you can actually pay. It does not get to sneak onto a free lane because that lane answered once last week. Last week is not a contract.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are relevant only as places you might run this gate, not as a quota, a hardware shape, a duration, or a promise that the lane stays up. This note does not name models or allotments. Those change, and a workflow that bakes in a number you did not re-read today will lie tomorrow. If you use that free access, keep it on probe-class envelopes. Send committed work to a lane whose failure you can explain to the person waiting.
The free server is a reasonable home for the gate process and for an append-only ledger. It is a poor home for the job that must finish before a launch review. A server you do not pay for can be restarted, contended, or withdrawn.
The ledger still has to survive that, or a restart will mint a second admission for the same attempt. Write the verdict before you open the socket. A crash between those two moments is an admission you may still owe. A crash before the verdict is an attempt that never existed.
Refuse in a test, not in production
The module below is a local proposal you can run on Python 3.9 or newer. It does not call a model. It decides whether a call is allowed to exist. Admission is cheaper than a transcript, and you can run the check on a laptop before you point it at any host.
from dataclasses import dataclass
from enum import Enum
class Lane(Enum):
PROBE = 'probe'
COMMITTED = 'committed'
class Decision(Enum):
ADMIT = 'admit'
REFUSE = 'refuse'
@dataclass(frozen=True)
class Envelope:
job_id: str
attempt: int
input_token_cap: int
output_token_cap: int
timeout_s: int
lane: Lane
has_deadline: bool
meter_named: bool
issued_at_s: int
@dataclass(frozen=True)
class Verdict:
decision: Decision
reason: str
PROBE_OUTPUT_CAP = 512
PROBE_TIMEOUT_S = 20
TICKET_TTL_S = 30
def admit(env: Envelope, now_s: int, seen: set[tuple[str, int]]) -> Verdict:
key = (env.job_id, env.attempt)
if key in seen:
return Verdict(Decision.REFUSE, 'attempt already admitted')
age = now_s - env.issued_at_s
if age < 0 or age > TICKET_TTL_S:
return Verdict(Decision.REFUSE, 'envelope expired or clock skew')
if env.attempt < 1:
return Verdict(Decision.REFUSE, 'attempt must start at 1')
if env.input_token_cap <= 0 or env.output_token_cap <= 0 or env.timeout_s <= 0:
return Verdict(Decision.REFUSE, 'caps and timeout must be positive')
if env.lane is Lane.PROBE:
if env.has_deadline or env.attempt > 1:
return Verdict(Decision.REFUSE, 'probe lane rejects deadlines and retries')
wide = env.output_token_cap > PROBE_OUTPUT_CAP or env.timeout_s > PROBE_TIMEOUT_S
if wide:
return Verdict(Decision.REFUSE, 'probe envelope too wide')
return Verdict(Decision.ADMIT, 'probe lane')
if not env.meter_named:
return Verdict(Decision.REFUSE, 'committed lane needs a named meter')
return Verdict(Decision.ADMIT, 'committed lane')
Read it as a bouncer, not as a scheduler. The bouncer does not care that another room is advertising a free hour. The bouncer cares that this ticket is fresh, unused, and matched to this door. The constants are your policy, not a vendor fact. If your logs show probes dying at eight seconds, do not keep twenty because a sample used twenty.
A retry mints a new envelope. Attempt two does not inherit attempt one. If the first attempt died on the probe lane, the second still has to qualify alone. That is how a timeout stops becoming a loop that spends the same prompt again. You can move attempt two onto the committed lane, name a meter, and widen the caps. You cannot stare at an idle free lane and call that a plan.
Three lines, three reasons
Save the module as admit.py. Run a probe that should pass, a retry that should die on the probe lane, and a committed job with no meter. You want three lines. A dashboard can wait until the reasons are boring.
python - <<'PY'
from admit import Envelope, Lane, Decision, admit
now = 1_700_000_000
seen = set()
cases = [
Envelope('job-7', 1, 800, 256, 15, Lane.PROBE, False, False, now),
Envelope('job-7', 2, 800, 256, 15, Lane.PROBE, False, False, now),
Envelope('job-9', 1, 4000, 1200, 60, Lane.COMMITTED, True, False, now),
]
for env in cases:
verdict = admit(env, now, seen)
print(env.job_id, env.attempt, verdict.decision.value, verdict.reason)
if verdict.decision is Decision.ADMIT:
seen.add((env.job_id, env.attempt))
PY
You should see the probe admitted, the retry refused, and the committed job refused because no meter is named. Rebuild attempt two on the committed lane with the meter flag set, and the gate admits it. That is the move. Change the lane, or change the envelope. Do not widen the free door because the caller is impatient.
When a refusal surprises you, debug the ticket before you debug the model. Print the issue time, the clock you passed in, and the attempt key. An expired envelope looks like a flaky provider if you only read the HTTP timeout.
A reused attempt key looks like a dropped response if you only read the client log. Append one ledger row per verdict: job id, attempt, lane, reason, both caps, and the clock. You can reconstruct a bad night from those rows without opening a completion.
Clock skew belongs in that row. The sample treats a negative age as a refuse, because a ticket from the future is not fresher. It is a lie about time.
If the host running the gate and the host stamping the envelope disagree by a minute, a thirty-second TTL refuses everything and you will blame the model. Stamp issued_at_s on the same host that calls admit, or sync the clocks first. Do not paper over the refuse by raising the TTL until the bug disappears. A long TTL is how a retry inherits a mood from an hour ago.
There is a second failure that looks like thrift. You set a tiny output cap, the gate admits the probe, and the client ignores the cap when it builds the request. The dataclass cannot save you from that.
Enforce the cap in the request builder. If the provider will not take a hard max, truncate the prompt and refuse to send when the counted input exceeds input_token_cap. A gate that only decorates a too-large call is theater.
Queue time is part of the ticket's age, not a free waiting room. If you enqueue first and admit later, the TTL expires in line and you have invented a refusal that feels like an outage. Admit, write the ledger, then enqueue. If the line is longer than the TTL, the job was never eligible for that lane. Lengthen the TTL only when you also lengthen what you are willing to owe if the worker dies mid-call. Time spent waiting is not a discount.
Who should walk away
Do not use this pattern for work that must complete. A probe lane has no deadline by construction. If the caller has a pager, a checkout, or a contract, the free lane is the wrong bet even when it looks empty. Empty is not reserved. A free lane is a place to learn the shape of a call, not a place to hide a launch.
Skip it when you cannot state an output cap. Letting the model decide how long to talk is an open tab, not an envelope. Skip it for inputs you have not chunked, such as a raw repository you hope will fit. The gate will admit a lie if you type a small cap over a huge prompt and then ignore the cap at call time.
The estimator can be wrong in both directions. A tight cap truncates a useful answer. A loose cap can still fit the lane and waste the attempt. This gate does not score quality. It only stops an attempt you did not mean to authorize. If you need evals, give them their own probe-class envelopes. Do not hide an evaluation suite inside a retry loop and call the suite free.
Nothing here measures a provider, names a model, or claims a lasting allotment. If free access shrinks, or the free server goes away, the policy should still refuse a deadline on the probe lane. The numbers in the sample are local. Change them to match the failure you have actually seen, not the failure a blog found convenient.
If you want a free server under the gate and free model access limited to probe-class envelopes, MonkeyCode is one place that currently offers both. Read the live terms before you depend on either. The gate still has to say no when the ticket does not fit.
Top comments (0)