A Token Budget Proxy for Shared Free LLM Tiers

Free LLM tiers are a shared resource with a hard ceiling, and the ceiling is usually measured in tokens, not requests. A single misconfigured batch job can consume a day's allowance in minutes, leaving every other user on the team with a 429 or a silent degradation. This article presents a lightweight proxy that assigns per-user token budgets and enforces them before a request reaches the upstream API. The implementation uses FastAPI and an in-memory store, and it is designed to sit in front of any OpenAI-compatible endpoint, including the free server offered by MonkeyCode.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a Token Budget Proxy?

When several developers share one free API key, the absence of per-user limits turns the quota into a tragedy of the commons. A background job that loops over a large dataset can exhaust the daily allowance in a few minutes, and the rest of the team only discovers the problem when their own requests start failing. For example, a team of five developers sharing a free tier with a 100,000-token daily limit might see one member run a data extraction job that consumes 80,000 tokens in a single hour. The remaining four members then have only 20,000 tokens for the rest of the day, which is barely enough for interactive development. With a per-user budget of 20,000 tokens, the extraction job would be stopped after the first 20,000 tokens, and the other four members would still have their full share.

The proxy solves this by moving the enforcement from the upstream provider to the edge, where the team can define its own fairness rules. It also provides a simple audit trail, because every request is logged with the user identifier and the token count.

The Architecture

The proxy is a thin HTTP layer that intercepts every chat completion request and performs three steps. First, it reads a user identifier from a custom header and loads that user's current budget from an in-memory store. Second, it estimates the token cost of the incoming request by combining a rough prompt heuristic with the requested max_tokens value. Third, if the estimated cost fits within the remaining budget, it forwards the request to the upstream API and updates the budget with the actual usage reported in the response.

Implementation

The core logic fits in a single FastAPI application. The example below uses a dictionary for storage, which is sufficient for a single-process deployment or a small team prototype.

import time
import httpx
from fastapi import FastAPI, HTTPException, Request, Response

app = FastAPI()
UPSTREAM_URL = "https://api.monkeycode.example/v1/chat/completions"
DAILY_QUOTA = 100_000  # tokens per user per day
budgets = {}

def estimate_tokens(text: str) -> int:
    # Rough heuristic: ~4 characters per token for English text.
    return max(1, len(text) // 4)

@app.post("/v1/chat/completions")
async def proxy(request: Request):
    user_id = request.headers.get("X-User-Id", "default")
    now = time.time()

    # Reset the daily budget for a new day.
    if user_id not in budgets or now > budgets[user_id]["reset"]:
        budgets[user_id] = {"used": 0, "reset": now + 86400}
    budget = budgets[user_id]

    body = await request.json()
    max_tokens = body.get("max_tokens", 512)
    prompt_text = "".join(m.get("content", "") for m in body.get("messages", []))
    estimated = estimate_tokens(prompt_text) + max_tokens

    if budget["used"] + estimated > DAILY_QUOTA:
        raise HTTPException(status_code=429, detail="Token budget exceeded")

    async with httpx.AsyncClient() as client:
        upstream = await client.post(UPSTREAM_URL, json=body, timeout=60.0)
        if upstream.status_code != 200:
            return Response(content=upstream.content, status_code=upstream.status_code)

        data = upstream.json()
        actual = data.get("usage", {}).get("total_tokens", estimated)
        budget["used"] += actual
        return Response(content=upstream.content, status_code=200, media_type="application/json")

The proxy reads the user ID from the X-User-Id header, which the caller must set. The token estimate is intentionally conservative, because it uses the full prompt length plus the maximum allowed completion length. The actual usage from the upstream response replaces the estimate whenever the usage field is present. The proxy also handles upstream errors gracefully, because a non-200 response is forwarded without consuming any tokens from the budget.

Configuration

The proxy exposes two configuration knobs: the daily quota and the reset interval. The DAILY_QUOTA constant controls the maximum number of tokens each user can consume in a rolling day, and the reset time is stored per user as a Unix timestamp. For a more flexible setup, the quota can be read from an environment variable or a configuration file, and the reset interval can be changed from 24 hours to a weekly window. The current implementation resets the budget lazily, which means the first request after the reset time starts a fresh budget.

Testing the Proxy

Start the server with uvicorn main:app --port 8000 and then run a few requests from two different users. The script below simulates a user exhausting the budget and receiving a 429.

for i in $(seq 1 5); do
  curl -s -X POST http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "X-User-Id: alice" \
    -d '{"messages":[{"role":"user","content":"Say hello"}],"max_tokens":10}'
  echo
done

Alternatively, a short Python script can verify the behavior programmatically.

import httpx

base = "http://localhost:8000/v1/chat/completions"
headers = {"X-User-Id": "bob"}
payload = {"messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 10}

for i in range(5):
    r = httpx.post(base, json=payload, headers=headers)
    print(i, r.status_code, r.text[:50])

If the daily quota is set to a small value, such as 50 tokens, the first request succeeds and the second one returns a 429. To test without a real upstream, point the UPSTREAM_URL to a local mock server that returns a fixed JSON response, which makes the test deterministic and independent of the free server's availability.

Limitations and Alternatives

The in-memory dictionary is the weakest part of this design, because it loses all budgets when the process restarts and it does not work across multiple instances. A production deployment should replace it with Redis or another shared store, and the reset logic should use a scheduled job instead of a lazy check. The token estimate is also a rough heuristic, and it can drift significantly from actual usage for non-English text or code. Finally, the proxy does not handle streaming responses, because it buffers the entire response before forwarding it, which defeats the purpose of streaming. Furthermore, the in-memory store is not thread-safe in a multi-worker deployment, so the proxy should be run with a single worker or use a shared store.

Several open-source API gateways, such as Kong or Traefik, offer rate limiting and quota enforcement out of the box. These tools are more robust than a custom proxy, but they require additional infrastructure and configuration. For teams that already run a gateway, adding a token-based quota plugin is often simpler than maintaining a separate service. The custom proxy is most valuable when the team needs a quick, lightweight solution that can be deployed as a single Python process.

Final Thoughts

A token budget proxy turns an invisible shared quota into a visible per-user allocation, and it prevents one runaway job from ruining the day for everyone. The implementation is small enough to read in a single sitting, and it can be extended with persistent storage, streaming support, and per-route quotas. For developers who want to experiment with this pattern, the MonkeyCode free server provides a convenient upstream target, and the project repository documents the current free-tier details. The proxy is not a replacement for a commercial gateway, but it is a practical first step toward responsible use of free LLM resources.

MonkeyCode provides free models that can run this workflow.