Ten Million Free Tokens Are a Balance, Not an Architecture

Last Thursday a support lead clicked Summarize on a thread with forty-one replies, waited six seconds, and got a summary that was maybe ninety percent right. The bad part arrived ninety minutes later, when that same button had been pressed four hundred times and the spend page showed a number I had never seen before. The request did not fail at the UI, and it did not fail inside the model either; it failed at the layer where nobody kept count. Three files imported a model client directly, each with its own retry policy, and none of them knew what the others had already spent.

So let me state the position this whole post hangs on: free tokens are a balance sheet, not an architecture. A grant of ten million tokens is genuinely useful, and I will show exactly where I spent mine, but generosity is not a design decision. What matters is what your service does at zero, and whether you can answer that question without opening four files and guessing.

The failure was never the model

When I traced the four hundred requests, the model had behaved fine; latency was acceptable and the outputs were mostly good. The failure was accounting, and accounting failures are boring in a way that makes them expensive: nothing crashes, nothing pages you, and the bill simply appears. I had treated a finite grant as ambient capacity, which is the same mistake as treating a free database tier as a schema.

Does that sound familiar? Almost every prototype I have reviewed this year has the same shape: a working demo, one environment file, and no single place where cost is decided. The model is the least interesting part of that stack, because the model can be swapped in an afternoon.

I ran the rewrite against MonkeyCode's free model access because I wanted the ledger to meet a real provider before I paid anyone, and the free server option gave me somewhere to run it that was not my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The offer I used, at the time of writing, advertises ten million free tokens plus a free server, and I am deliberately not going to promise you a latency number or a hardware profile, because those are the parts that change.

One door, and a test that proves there is only one

My rule is unglamorous: exactly one function in the codebase is allowed to touch a provider. Everything else calls it. That function reserves budget before the network call, commits real usage after it, and releases the reservation when the call throws.

# app/gateway.py
from __future__ import annotations
from dataclasses import dataclass
from typing import Protocol

class BudgetExceeded(Exception):
    """Raised before the network call, never after it."""

@dataclass(frozen=True)
class Usage:
    prompt_tokens: int
    completion_tokens: int

    @property
    def total(self) -> int:
        return self.prompt_tokens + self.completion_tokens

class Provider(Protocol):
    name: str
    def complete(self, model: str, prompt: str, max_tokens: int) -> tuple[str, Usage]: ...

The ledger is a single row per account, because a row is something you can update atomically and a spreadsheet is not. Reserve is a conditional update, so two concurrent requests cannot both squeeze past the ceiling.

# app/ledger.py
RESERVE = """
update budget set reserved = reserved + :est
 where account_id = :acct
   and reserved + spent + :est <= granted
"""
COMMIT = """
update budget set reserved = reserved - :est, spent = spent + :actual
 where account_id = :acct
"""
RELEASE = """
update budget set reserved = reserved - :est
 where account_id = :acct and reserved >= :est
"""

class Ledger:
    def __init__(self, conn):
        self.conn = conn

    def reserve(self, acct: str, est: int) -> None:
        cur = self.conn.execute(RESERVE, {"acct": acct, "est": est})
        if cur.rowcount != 1:
            raise BudgetExceeded(f"{acct}: cannot reserve {est} tokens")

    def commit(self, acct: str, est: int, usage: Usage) -> None:
        self.conn.execute(COMMIT, {"acct": acct, "est": est, "actual": usage.total})
        self.conn.commit()

    def release(self, acct: str, est: int) -> None:
        self.conn.execute(RELEASE, {"acct": acct, "est": est})
        self.conn.commit()

With that in place the handler becomes short enough to read in a review, and the ordering is the point: estimate, reserve, call, commit, or release.

def estimate(text: str, max_tokens: int = 400) -> int:
    # ~4 characters per token, plus headroom for the completion
    return -(-len(text) // 4) + max_tokens

def summarize(acct: str, text: str) -> str:
    est = estimate(text)
    ledger.reserve(acct, est)
    try:
        out, usage = provider.complete(model, text, max_tokens=400)
    except Exception:
        ledger.release(acct, est)
        raise
    ledger.commit(acct, est, usage)
    return out

The test I actually care about is not the happy path, it is the one that proves a failed call costs nothing and that an exhausted account never reaches the wire.

class Fake:
    name = "fake"
    def __init__(self, usage=None, boom=False):
        self.calls, self.usage, self.boom = 0, usage, boom
    def complete(self, model, prompt, max_tokens):
        self.calls += 1
        if self.boom:
            raise TimeoutError("upstream stalled")
        return "ok", self.usage

def test_timeout_releases_reservation(db):
    led = Ledger(db)
    with pytest.raises(TimeoutError):
        run(led, Fake(boom=True), acct="team-a", text="hello")
    row = db.execute("select reserved, spent from budget where account_id='team-a'").fetchone()
    assert row == (0, 0)

def test_exhausted_account_never_calls_provider(db):
    led = Ledger(db)
    fake = Fake(usage=Usage(10, 10))
    with pytest.raises(BudgetExceeded):
        run(led, fake, acct="team-empty", text="x" * 10_000)
    assert fake.calls == 0

Reserve, then commit, then argue about the estimate

My estimate is wrong in both directions, and I know it. Character division undercounts code and JSON badly, so a prompt full of stack traces can blow past the reservation. Overcounting is the cheaper failure: you stop early, you leave tokens on the table, and nobody is surprised by a bill.

When actual usage exceeds the reservation, commit still writes the real number, which means the row can temporarily go negative in reserved and the next reserve fails until you reconcile. I accept that, because a brief refusal is better than a silent overspend. A nightly job compares provider-side usage to spent; if the gap exceeds two percent, the model or the tokenizer changed under me, and I want that alarm.

I also keep the only-door rule executable, because a comment is not a policy:

! grep -rn "import openai\|from anthropic\|ai-sdk" app/ --include="*.py" | grep -v "app/gateway.py"

That line runs in CI and fails the build. It has caught two well-meaning refactors already, both of which "just needed a quick call" in a route handler.

What the free server actually buys you

The free server is not interesting because it is free; it is interesting because it is a real deploy target you can break without a meeting. I use it for the branch that runs against a free model, so the ledger gets exercised by a container with the same environment variables as production.

The part that keeps that box alive is a boot check, not a retry wrapper. If the base URL, the model id, or the ceiling is missing, the process should refuse to start rather than serve traffic with an unbounded budget.

import os, sys
REQUIRED = ("PROVIDER_BASE_URL", "PROVIDER_MODEL", "TOKEN_CEILING")
missing = [k for k in REQUIRED if not os.getenv(k)]
if missing:
    sys.exit(f"refusing to boot, missing: {missing}")

Then a health endpoint that reports the remaining balance, which turns deployment into a one-liner you can run after every push.

curl -s "$BASE_URL/healthz" | jq '.remaining_tokens, .provider'

If that number moves while you are not clicking anything, something is calling the model on page load, and you have found your bug without reading a log.

When a grant is enough, and when it is not

Situation	Free grant is fine	Start paying instead
Internal tool, few users	Yes, with a per-account ceiling	When several teams share one balance
Customer-facing latency	No, cold starts will show	When p95 matters more than price
Regulated or personal data	No, until you have a data path review	Always
Batch backfills	Yes, at low priority	When a failed batch costs a day

I would not run anything with contractual uptime, medical text, or a per-tenant data-residency clause on a free tier, and I would not use this pattern as an excuse to skip a real budget owner. If your team cannot name the person who approves a spend increase, the free grant is hiding an org problem rather than solving a technical one.

What the balance sheet buys you is optionality: when the grant runs out you change one file, not forty. So here is my question for you, and I want the ugly version rather than the polished one: which handoff in your AI feature is least stable right now, and what exact failure do you see there, a 402 from the provider, a 504 from your own gateway, or a 200 with an empty string in it?