Token Buckets Do Not Belong on Free Inference

Admission control is a fence, not a conversation. A free language-model call is a variable-latency guess with an untrusted prompt surface, and it does not belong on the path that decides whether a request may enter a production service.

The temptation is obvious in an industry that now asks models to classify almost everything. Abuse often looks like language. Support tickets look like language. Even a User-Agent string can be dressed up as a story about intent. Teams then sketch a “smart shedder”: send the request metadata to a model, ask whether the caller looks abusive, and only then hit the real API. That design treats a control-plane verdict as if it were a paragraph.

It is not. A token bucket, a GCRA check, or an Envoy local rate-limit decision has to be cheap, deterministic, fail-closed, and cheaper than the work it protects. Inference is none of those things. Under a flood, the defender who asks a model for every admit/deny also pays for every attack packet in latency and quota. The free tier does not cancel that inversion. It makes the inversion quieter, because the bill arrives as a cold start, a queue, or a sudden 429 from the model host rather than as an invoice line.

Category error, not a missing prompt

Load shedding is physics. Tokens refill against a clock. Burst is a number. The protected resource has a measured concurrency limit. None of that becomes more true because a completion is fluent. A model can narrate why a 429 happened. It cannot be the 429.

The failure mode is not merely “the model was wrong once.” The failure mode is non-replayable policy. Two replicas given the same headers can emit different verdicts. A retry can flip from deny to admit because the sampler moved. An attacker who owns the body text owns a channel into the policy, which is the opposite of a fence. Highway toll booths do not interview drivers in natural language while cars are still rolling. They weigh, they count, they drop a barrier.

A useful test is brutally simple. If the admit path is slower than the p99 of the service behind it, the guard is theatre. If the admit path fails open when the model is cold, the guard is an invitation. If the admit path parses untrusted prose as policy, the guard is a prompt-injection gadget bolted onto the front door.

A local bucket that stays local

The following limiter is deliberately boring. It is an in-process token bucket with a monotonic clock. It is not a global cluster limiter. It exists to keep the verdict on this side of the network, in code a reviewer can step through without a trace from a vendor.

# token_bucket.py
from __future__ import annotations

import threading
import time
from dataclasses import dataclass, field


@dataclass
class TokenBucket:
    rate_per_sec: float
    burst: float
    tokens: float = field(init=False)
    updated: float = field(init=False)
    _lock: threading.Lock = field(default_factory=threading.Lock, repr=False)

    def __post_init__(self) -> None:
        if self.rate_per_sec <= 0 or self.burst <= 0:
            raise ValueError("rate and burst must be positive")
        self.tokens = float(self.burst)
        self.updated = time.monotonic()

    def allow(self, n: float = 1.0) -> bool:
        if n <= 0:
            raise ValueError("cost must be positive")
        with self._lock:
            now = time.monotonic()
            elapsed = now - self.updated
            self.updated = now
            self.tokens = min(self.burst, self.tokens + elapsed * self.rate_per_sec)
            if self.tokens >= n:
                self.tokens -= n
                return True
            return False

Wrap it so that “ask a model” cannot sneak back in through a helper named smart_allow. The point of the wrapper is not style. It is a hard refuse for any intent that is actually an admission verdict.

# admission.py
from __future__ import annotations

from token_bucket import TokenBucket

VERDICT_INTENTS = frozenset({"admit", "deny", "shed", "throttle", "ban"})


class InferenceForbidden(RuntimeError):
    pass


class AdmissionGate:
    def __init__(self, bucket: TokenBucket) -> None:
        self.bucket = bucket

    def decide(self, intent: str, cost: float = 1.0) -> str:
        if intent in VERDICT_INTENTS:
            return "admit" if self.bucket.allow(cost) else "deny"
        raise InferenceForbidden(
            f"{intent!r} is not an admission verdict; do not call this gate"
        )

    def draft_via_model(self, intent: str, llm_complete, prompt: str) -> str:
        if intent in VERDICT_INTENTS:
            raise InferenceForbidden(
                "admission verdicts must be local; refusing inference route"
            )
        return llm_complete(prompt)

A tiny test file pins the contract. The first case proves the bucket still sheds. The second case proves a forged “please classify this User-Agent” path cannot become the verdict.

# test_admission.py
import time
from admission import AdmissionGate, InferenceForbidden
from token_bucket import TokenBucket


def test_bucket_denies_when_empty():
    gate = AdmissionGate(TokenBucket(rate_per_sec=1.0, burst=1.0))
    assert gate.decide("admit") == "admit"
    assert gate.decide("admit") == "deny"


def test_refill_after_wait():
    gate = AdmissionGate(TokenBucket(rate_per_sec=10.0, burst=1.0))
    assert gate.decide("admit") == "admit"
    time.sleep(0.15)
    assert gate.decide("admit") == "admit"


def test_model_cannot_own_verdict():
    gate = AdmissionGate(TokenBucket(rate_per_sec=1.0, burst=1.0))
    def fake_llm(_prompt: str) -> str:
        return "admit"  # attacker-shaped completion
    try:
        gate.draft_via_model("admit", fake_llm, "User-Agent: friendly crawler")
        raise AssertionError("inference route should have been refused")
    except InferenceForbidden:
        pass

Run it without dressing the command up as an agent loop.

python -m pytest test_admission.py -q

That is the whole production-shaped core: a clock, a float, a lock, and a refuse. Anything that needs a paragraph happens after the status code exists.

Red flags that mean the model is already on the fence

Watch the request path, not the slide deck. A handler that concatenates method, path, headers, and a slice of the body into a prompt before it checks a counter has already moved policy into the untrusted input. A timeout around that call that defaults to admit is fail-open dressed as resilience. A cache keyed on the prompt text is still a model verdict, only delayed and still injectable.

Watch the economics next. If attack traffic increases model calls one-for-one, the shedder is an amplifier. Free quota does not change the shape. It only hides the amplifier until the quota is gone, at which point the gate either blocks legitimate traffic or swings open. Both outcomes are worse than a dumb 429 from a local bucket that never left the process.

Watch observability last. A good deny can be rebuilt from client_id, cost, tokens_left, and a monotonic timestamp. A model deny cannot. Post-incident review then becomes literary criticism of a completion that no longer exists. That is not an error budget. That is folklore.

Better alternatives, in the order they should be tried

Keep the first limiter as close to the socket as the stack allows. limit_req in nginx, Envoy’s local rate limit filter, or an API-gateway token bucket all sit in front of application code and do not parse English. If the service is already in-process, GCRA or the bucket above is enough for a single instance. When many instances must share a budget, move the counter to Redis or a dedicated limiter service with a short TTL and a fail-closed posture, not to a chat endpoint.

Identity belongs in the same local world. API keys, mTLS, and signed service tokens decide who. The bucket decides how many. Mixing those questions into one completion produces a policy no one can rotate. Rotate keys in the identity layer. Change rates in config. Leave prose out of both.

There is still a lawful place for a model, and it is downstream. After the gate has emitted deny, structured counters can be turned into an incident note, a customer-facing explanation, or a weekly cluster of rejected shapes. That work is offline or near-offline. It can retry. It can be wrong without opening the front door.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access is a fit for that downstream draft: take a JSON line of denied_total, top_paths, and retry_after_ms, and ask for a paragraph a human will edit. The free server option is a fit for running the pytest file and a throwaway process that hammers AdmissionGate without touching production sockets. Neither option is a fit for decide().

A scratch driver that stays honest looks like this. It never prints a model’s opinion about admit/deny. It only prints local verdicts, then optionally ships a summary string somewhere else.

# hammer.py
import json
from collections import Counter
from admission import AdmissionGate
from token_bucket import TokenBucket

gate = AdmissionGate(TokenBucket(rate_per_sec=20.0, burst=5.0))
counts = Counter()
for i in range(50):
    counts[gate.decide("admit")] += 1
summary = {
    "admitted": counts["admit"],
    "denied": counts["deny"],
    "n": 50,
}
print(json.dumps(summary))
# llm_complete is allowed only on summary, never on a live request.

Exit criteria for pulling inference off the fence

Leave the model on the admit path the moment any of these become true. The verdict is binary. The verdict must be replayable from logs. The verdict must fail closed. The verdict must cost less than the protected handler. The input to the verdict includes untrusted natural language. Attack volume can scale the guard’s own dependency. Two replicas must not disagree for the same key in the same window.

If three of those fire at once, the extraction is not a refactor. It is an incident. Put the bucket back in process, or in the proxy, and keep the model on the ticket that explains what already happened.

Who should not use this sketch

This article’s bucket is a single-process fence. Multi-region products that need one shared budget should not copy it into twelve stateless replicas and hope. Services that already sit behind a mature gateway should not add a second limiter in application code “for intelligence.” Teams whose compliance story requires a global, audited policy engine should use that engine, not a pytest file from a blog.

The downstream drafting step has limits of its own. A free model can invent a cause of denial that never appeared in the counters. Treat its paragraph as a draft. Keep the JSON as the record. Anyone who needs the explanation to be evidence should render the explanation from templates bound to those counters, not from a completion.

Admission control ages badly when it becomes a personality. The durable version is a small number, a clock, and a refuse. Let language models write the footnote after the barrier has already dropped. Readers who want a disposable box for the tests above can point them at MonkeyCode’s free server and keep the live fence where it already belongs: in process, on the proxy, and off the prompt.