The Spend-Curve Test: Free Token Allowance or Self-Hosted, Decided With a Replay Harness

Most developers compare a free token allowance with a self-hosted box by lining up monthly prices, and that comparison quietly answers the wrong question. The number you actually need is the shape of your cumulative spend, because a fixed allowance survives a steady workload and evaporates under a bursty one. So here is my conclusion up front: replay your own prompt set, plot tokens against the allowance, and let the crossover day make the hosting decision for you. Everything below is a decision guide around one small, runnable harness — roughly twenty minutes of work, and no benchmark folklore required.

1. The question a price tag cannot answer

A free allowance is a fixed budget of tokens, and budgets do not fail linearly. One afternoon of agent retries can eat a week of your planned headroom, and then you are debugging a 429 instead of shipping. Self-hosting has the opposite failure mode: a fixed hardware bill that is generous on idle days and punishing when your queue is empty. Neither failure is visible in a per-token price, which is why so many "free vs paid" comparisons collapse the moment real traffic arrives. What we want is a curve, and curves are cheap to measure.

2. Build the spend-curve harness in four steps

The harness replays a JSONL file of prompts against any OpenAI-compatible chat endpoint and records the usage block from each response. It uses only the Python standard library, so there is nothing to install and nothing to pin.

Step 1 — describe your workload as prompts, with a simulated day per line. The day field lets you model bursty traffic instead of pretending every day looks the same.

{"day": 1, "id": "summarize-pr-1", "prompt": "Summarize this diff in five bullets: ..."}
{"day": 1, "id": "summarize-pr-2", "prompt": "Summarize this diff in five bullets: ..."}
{"day": 4, "id": "refactor-call", "prompt": "Rewrite this function to accept a cancellation token: ..."}

Step 2 — write the replay script. It stops on the first 429 rather than retrying around a rate limit, because a retry loop turns a measurement into a traffic incident.

#!/usr/bin/env python3
"""spend_curve.py - replay prompts, record tokens, report the crossover day."""
import csv, json, os, sys, urllib.error, urllib.request

BASE = os.environ["BASE_URL"].rstrip("/")
KEY = os.environ.get("API_KEY", "")
MODEL = os.environ["MODEL"]
ALLOWANCE = int(os.environ.get("ALLOWANCE", "0"))


def call(prompt: str, timeout: int = 90) -> dict:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 512,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.load(resp)


def tokens(payload: dict):
    usage = payload.get("usage")
    if not usage:
        return None  # fail loudly; never guess a metered number
    return usage["prompt_tokens"] + usage["completion_tokens"]


cumulative, rows, crossover = 0, [], None
with open(sys.argv[1]) as fh:
    for line in fh:
        item = json.loads(line)
        try:
            payload = call(item["prompt"])
        except urllib.error.HTTPError as err:
            if err.code == 429:
                print(f"rate limited at {item['id']}; stopping", file=sys.stderr)
                break
            raise
        spent = tokens(payload)
        if spent is None:
            sys.exit("endpoint returned no usage block; use a metered route")
        cumulative += spent
        rows.append([item["day"], item["id"], spent, cumulative])
        if crossover is None and ALLOWANCE and cumulative > ALLOWANCE:
            crossover = item["day"]

with open("spend_curve.csv", "w", newline="") as fh:
    csv.writer(fh).writerows([["day", "request", "tokens", "cumulative"], *rows])
print(f"total={cumulative} allowance={ALLOWANCE} crossover_day={crossover}")

Step 3 — run it against each candidate route with identical inputs. Same prompt file, same output ceiling, so the only variable left is the endpoint itself.

BASE_URL=https://your-endpoint.example/v1 API_KEY=... MODEL=... \
ALLOWANCE=10000000 python3 spend_curve.py prompts.jsonl

Step 4 — sanity-check the meter before you trust the curve. Send three identical prompts and confirm the token counts are stable; if they wobble, your template is injecting something you did not intend. A second caveat worth knowing: some newer models want max_completion_tokens instead of max_tokens, and some gateways silently ignore both. Read the endpoint's documentation and reconcile the usage block against your own estimate before drawing conclusions.

3. Read the curve: three shapes, three decisions

Once you have spend_curve.csv, aggregate by day and look at the slope rather than the total. I care about three shapes, and each one points somewhere different.

• Steady slope, low daily volume: the allowance lasts for months, and self-hosting is pure overhead you would be paying for out of pride.
• Steady slope, high volume: crossover arrives inside your planning window, so you need a costed second route designed before day one.
• Spiky slope with long idle gaps: the allowance is either plenty or gone by lunchtime, and a capped queue is safer than either extreme.

Turning that into a fit decision gives a table I would genuinely paste into a design document, because it forces the tradeoffs into the open.

Signal	Free allowance fits when	Self-hosted fits when	Paid hosted fits when
Volume vs budget	measured crossover lands past your planning horizon	you keep the box busy most days	demand is bursty and ops appetite is low
Data residency	prompts carry no regulated data	data cannot leave your network	vendor terms are already approved
Latency tail	p95 is acceptable for interactive use	you can tune and own the tail	you need a contracted SLA
Ops capacity	nobody on the team wants to patch drivers	someone owns the machine and the on-call	you would rather buy uptime
Iteration speed	you are still rewriting prompts daily	the workload is stable and understood	the workload changes but budget exists
Exit cost	the API is OpenAI-compatible and swappable	hardware holds resale value	the contract term is short

4. Where a free tier actually earns its place

The operator of the project I used for this workflow describes two free-tier options: free model access with a stated allowance of ten million tokens, and a free server option for running work off your laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat both as availability claims to verify on the project's own page rather than permanent guarantees, because free tiers move and allowances get adjusted. What makes them useful for this harness is narrower and more honest: the free route is the cheapest place to collect your first spend curve, and the free server is a reasonable place to park the nightly replay job.

Does that mean you should build a production path on a free allowance? No, and I would push back on anyone who says so without a measured crossover day. The free route is a measuring instrument and a prototyping surface; the moment your curve crosses the allowance in week one, you have learned something more durable than a discount.

5. Limitations and who should skip this

This harness measures tokens, not output quality, so pair it with a small acceptance check before you trust any route with real work. It also assumes your endpoint returns a usage block, which some proxies strip; without that block you are estimating, and estimates should not drive a hosting decision. Finally, the curve is only valid for the prompt set you replayed — change the template and the number changes with it, which is why the JSONL file belongs in the repository.

Skip this approach if a compliance rule forbids third-party inference entirely, because then the decision is already made and the curve is trivia. Skip it too if your workload is one predictable daily batch that fits on a napkin, or if you cannot spare half a day to write realistic prompts, since a curve built from toy inputs is worse than no curve at all.

If you do run it, commit the CSV next to the decision it justified. Six weeks from now, when someone asks why the agent runs where it runs, that file is the answer — and if you want a zero-cost place to generate the first version of it, the free route is a sensible starting line.