Free Model Tokens Are a Hard Budget. Enforce Them in Code, Not in Discipline.

Last month a colleague showed me a demo that burned a free model allowance in one afternoon, and nobody noticed until the quota was gone. A background job that should have summarized ten documents kept re-summarizing the same ten in a retry loop, and every iteration quietly spent tokens that looked free. The model access was free, the server was free, and the architecture had no idea how much it was consuming. That is the moment I stopped treating free infrastructure as a gift and started treating it as a contract.

Here is the position I want to defend: free tokens are a hard budget, and your code should enforce that budget before your discipline does. Most teams treat the free allowance as a trial period and the paid plan as the real constraint, which is exactly backwards. The free allowance is the tightest ceiling you will ever have, and if your feature cannot survive it, paying for more tokens only buys you more time to be sloppy. The same logic applies to a free server, which will restart, lose state, and run out of disk while you are asleep.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open source project that offers free model access and a free server option, and the free allowance I am working with here is ten million tokens, which is generous enough to hide every design flaw you have. That generosity is precisely why you need enforcement before you need features, so let me show you the enforcement.

I built a small budget proxy to prove the point, and the first version failed in a predictable way. It checked the daily budget before forwarding and recorded the actual usage after the response, which sounds reasonable until two requests pass the check at once. A burst of parallel calls can overshoot the ceiling by the size of the burst, because nothing reserves the tokens between the check and the spend. The fix is the same pattern you use for inventory or seats: reserve first, settle later.

# token_budget_proxy.py
import asyncio
import sqlite3
import time
from datetime import date

import httpx
from fastapi import FastAPI, Request, Response

app = FastAPI()
UPSTREAM = "https://api.monkeycode.example/v1/chat/completions"  # replace with your endpoint
DAILY_BUDGET = 10_000_000  # operator-supplied free allowance
DB_PATH = "budget.db"
_lock = asyncio.Lock()


def init_db() -> None:
    with sqlite3.connect(DB_PATH) as conn:
        conn.execute(
            """
            CREATE TABLE IF NOT EXISTS usage (
                day TEXT NOT NULL,
                route TEXT NOT NULL,
                reserved INTEGER NOT NULL,
                actual INTEGER NOT NULL DEFAULT 0,
                ts REAL NOT NULL
            )
            """
        )
        conn.commit()


def spent_today() -> int:
    today = date.today().isoformat()
    with sqlite3.connect(DB_PATH) as conn:
        row = conn.execute(
            """
            SELECT COALESCE(SUM(
                CASE WHEN actual > 0 THEN actual ELSE reserved END
            ), 0)
            FROM usage WHERE day = ?
            """,
            (today,),
        ).fetchone()
    return row[0]


@app.post("/v1/chat/completions")
async def chat(request: Request) -> Response:
    body = await request.body()
    estimate = len(body) // 4  # rough input token estimate

    async with _lock:
        if spent_today() + estimate >= DAILY_BUDGET:
            return Response(
                content='{"error":"daily_token_budget_exhausted"}',
                status_code=429,
                media_type="application/json",
            )
        with sqlite3.connect(DB_PATH) as conn:
            cur = conn.execute(
                "INSERT INTO usage (day, route, reserved, ts) VALUES (?, ?, ?, ?)",
                (date.today().isoformat(), request.url.path, estimate, time.time()),
            )
            row_id = cur.lastrowid
            conn.commit()

    async with httpx.AsyncClient(timeout=60) as client:
        upstream = await client.post(
            UPSTREAM,
            content=body,
            headers={"content-type": "application/json"},
        )

    payload = upstream.json()
    actual = payload.get("usage", {}).get("total_tokens", estimate)
    with sqlite3.connect(DB_PATH) as conn:
        conn.execute(
            "UPDATE usage SET actual = ? WHERE rowid = ?",
            (actual, row_id),
        )
        conn.commit()

    return Response(
        content=upstream.content,
        status_code=upstream.status_code,
        media_type="application/json",
    )


init_db()

Two decisions in that proxy matter more than the rest. The asyncio lock serializes the check and the reservation, so concurrent requests see the same ledger instead of racing past it, while the settlement replaces the estimate with the real usage. If the upstream call fails, the reservation stays in the ledger, which is intentional because retries are exactly what burned my colleague's allowance. A budget that counts failed attempts is a budget that stops retry loops.

Run it with uvicorn token_budget_proxy:app --port 8000, point your app at it instead of the upstream endpoint, and set DAILY_BUDGET to something small like ten thousand tokens to watch the 429 appear. Then raise the budget to the real allowance and let the proxy be the single place where cost becomes visible. What does your client do when the proxy returns 429? If the answer is retry harder, you have just found your first bug. The proxy is deliberately boring, because the last thing you want in a cost control path is cleverness.

# burn_test.py
import asyncio

import httpx

PROXY = "http://localhost:8000/v1/chat/completions"


async def one_request(client: httpx.AsyncClient, i: int) -> None:
    payload = {
        "messages": [
            {"role": "user", "content": f"Summarize document {i} in ten sentences."}
        ]
    }
    response = await client.post(PROXY, json=payload)
    usage = response.json().get("usage", {}).get("total_tokens", 0)
    print(i, response.status_code, usage)


async def main() -> None:
    async with httpx.AsyncClient() as client:
        await asyncio.gather(*[one_request(client, i) for i in range(20)])


asyncio.run(main())

The burn test sends twenty parallel requests, and the interesting output is not the responses but the usage table afterward. You will see reserved values that are higher than actual values, because the reservation is a rough estimate and the settlement is the truth. That conservatism is intentional, and it is the difference between a budget that protects you and a budget that merely reports your regrets.

# usage.py
import sqlite3

with sqlite3.connect("budget.db") as conn:
    rows = conn.execute(
        """
        SELECT day, route, SUM(reserved), SUM(actual), COUNT(*)
        FROM usage
        GROUP BY day, route
        """
    ).fetchall()

for day, route, reserved, actual, count in rows:
    print(day, route, reserved, actual, count)

The usage table is the part your finance person will actually read. It shows the day, the route, what you reserved, what you really spent, and how many requests slipped through. If the reserved column is consistently double the actual column, your estimate is too crude and you are leaving budget on the table.

Who should not use this approach? If you are running one-off experiments with no users and no background jobs, a budget proxy is overhead you do not need. If you need sub-millisecond overhead, the sqlite write on every request will annoy you, so batch the records or keep a counter in memory and flush it periodically. If you scale the proxy horizontally, the asyncio lock and the local sqlite file stop coordinating across processes, and you need a shared store like Postgres or Redis. The proxy is a single-process tool, and pretending otherwise is how budgets leak.

Here is the checklist you will forget by Friday:

• Put the budget check before the upstream call, never after it.
• Reserve an estimate before you spend, then settle with the real usage from the response.
• Return 429 with a structured error, and make your client treat it as a stop signal, not a retry trigger.
• Run the proxy in CI with a tenth of the real allowance, so the 429 becomes an assertion instead of a surprise.

So here is my answer to the question nobody asks until the quota is gone: free infrastructure is not a perk, it is a production constraint wearing a marketing hat. Build the enforcement before you build the feature, keep the enforcement boring, and let the free allowance tell you the truth about your architecture. If your feature cannot live inside ten million tokens and a server that forgets everything, is the free tier really the problem? If you have a free allowance burning a hole in your repo, the proxy above is a ten minute start.