Migration Diary: Prove the Free Endpoint Can Hold Your Token Load Before Cutover

You should not cancel the paid key until an offline projection shows your real traffic fits the free allowance with slack left over. This entry is the fifth step in my migration diary, and it covers the part everyone skips: proving the token math before the cutover window opens. The artifact below is a small, dependency-free Python gate that reads your existing usage logs, projects a full month, and fails loudly when the headroom is too thin. If it passes, you move on to the leftovers inventory; if it fails, you keep paying and you know exactly why.

Most teams measure tokens by reading a dashboard once, dividing by days, and multiplying by thirty. That estimate quietly assumes the new endpoint tokenizes text the same way the old one did, which is rarely true. A five to fifteen percent drift on the input side is common, and it is enough to push a tight budget over the edge mid-month. Measure the drift instead of assuming it, then feed the measured number into the projection.

Step 1 — Freeze a usage corpus from logs you already have

Start with request-level records, not monthly totals, because you need a distribution rather than an average. Any structured log line that carries prompt and completion token counts will do, and you only need fourteen to thirty days of history. Export to JSONL so the gate can stream records without loading everything into memory.

# From an NDJSON log stream that carries a usage object per request
jq -c 'select(.usage != null)
  | {day: .timestamp[0:10],
     input_tokens: .usage.prompt_tokens,
     output_tokens: .usage.completion_tokens}' \
  usage.ndjson > corpus.jsonl

Two details matter here. First, keep the day field, because the gate needs the number of distinct observed days to compute a daily rate; second, drop records where usage is null instead of substituting zeros. Zero-filled rows will drag your daily rate down and hand you a falsely optimistic projection.

Step 2 — Measure tokenizer drift with a small probe

Before projecting, send a fixed sample of real prompts to the target endpoint and compare its reported counts against your local estimate. Thirty to fifty prompts is enough to get a usable multiplier, and you should sample across your longest prompts rather than the average ones. Record the ratio per request and take the high percentile, not the median, because your tail is what breaks the budget.

# probe_drift.py (sketch — wire in your own client call)
from statistics import quantiles

ratios = []
for prompt in sample_prompts[:50]:
    local_tokens = len(prompt.split())          # placeholder local estimate
    reported = call_target(prompt)               # returns input token count
    ratios.append(reported / local_tokens)

ratios.sort()
drift = quantiles(ratios, n=20)[-1]              # p95 multiplier
print(f"apply drift multiplier: {drift:.3f}")

Use the p95 multiplier, not the mean, when you feed --drift into the gate. If your local estimate is good, the multiplier lands near 1.0 and the whole step costs you a few cents and ten minutes. If it lands at 1.2, that is not noise; that is the reason your first week on a free tier would have ended early.

Step 3 — Run the budget gate

This script is deliberately offline: it makes no network calls, needs no SDK, and can run in CI against a committed corpus. It exits 0 when the projection fits with your required headroom, 1 when it does not, and 2 when the corpus is empty so you never get a silent pass. Keep the allowance as a flag rather than a constant so a pricing change is a one-line diff.

#!/usr/bin/env python3
"""budget_gate.py - offline token-budget gate for an AI endpoint cutover."""
from __future__ import annotations

import argparse
import json
import statistics
import sys


def load_corpus(path: str) -> list[dict]:
    records = []
    with open(path, encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                raw = json.loads(line)
                records.append({
                    "day": str(raw["day"]),
                    "tokens": int(raw["input_tokens"]) + int(raw["output_tokens"]),
                })
            except (KeyError, TypeError, ValueError) as exc:
                sys.exit(f"{path}:{line_number}: bad record ({exc})")
    return records


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("corpus")
    parser.add_argument("--allowance", type=int, default=10_000_000)
    parser.add_argument("--days", type=int, default=30)
    parser.add_argument("--headroom", type=float, default=0.20)
    parser.add_argument("--drift", type=float, default=1.0)
    args = parser.parse_args()

    records = load_corpus(args.corpus)
    if not records:
        print("empty corpus: refusing to project a budget from nothing")
        return 2

    observed_days = len({record["day"] for record in records})
    total = sum(record["tokens"] for record in records)
    daily = total / observed_days * args.drift
    projected = daily * args.days
    per_request = sorted(record["tokens"] for record in records)
    p95 = statistics.quantiles(per_request, n=20)[-1] if len(per_request) >= 20 else per_request[-1]
    ceiling = args.allowance * (1 - args.headroom)

    print(f"observed days      : {observed_days}")
    print(f"requests           : {len(per_request)}")
    print(f"tokens (recorded)  : {total:,}")
    print(f"drift multiplier   : {args.drift:.3f}")
    print(f"projected {args.days}d     : {projected:,.0f}")
    print(f"budget ceiling     : {ceiling:,.0f}")
    print(f"p95 request tokens : {p95:,.0f}")

    if projected > ceiling:
        print("FAIL: projected load does not fit with the requested headroom")
        return 1
    print("PASS: projected load fits")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it three times with different headroom values to see how brittle the fit is: twenty percent, ten percent, and zero. A workload that passes at zero but fails at ten is not ready, because one retry storm or one batch backfill will consume the slack you convinced yourself you had.

python3 budget_gate.py corpus.jsonl --drift 1.14 --headroom 0.20

Step 4 — Inventory the leftovers before you delete anything

Cutover is not the moment you stop calling one URL and start calling another; it is the moment you find out which side systems were quietly pinned to the old provider. Walk this table with your own repository open, and mark each row honestly rather than aspirationally.

Leftover	Symptom if ignored	Action
Prompt-shim layer	Prompts tuned to one model degrade on another	Keep the shim, rewrite its rules
Retry and backoff config	Backoff tuned to paid latency hammers a shared server	Re-derive from new error mix
Eval baseline	Scores are pinned to the old model's output	Re-record with labelled provenance
Cost and usage dashboard	Alerts fire on a key you no longer hold	Repoint at the new usage source
Prompt cache warmer	Budget spent warming a cache nobody reads	Delete it
Fallback route	Silent failover resurrects the cancelled key	Remove the credential, not just the call

The last row is the one that bites hardest in my experience, because a dormant fallback path can sit unreferenced for weeks and then reactivate during an incident. Remove the credential itself rather than the code path that uses it, so a stale branch cannot spend money you thought you had stopped spending.

Where a free tier fits into this cutover

This is the point in the diary where the tooling matters, so here is the honest version. MonkeyCode offers free model access and a free server option, and the operator of this account states the free tier includes 10 million tokens; treat that figure and the server option as claims to re-verify on the vendor's pricing page on the day you plan the cutover, not as constants you can bake into a three-month roadmap. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The reason the gate above exists is that a fixed allowance converts a soft cost problem into a hard capacity problem. With metered billing, a traffic spike costs more money; with a fixed allowance, a traffic spike costs you availability, and availability failures are much harder to explain to users than a larger invoice. The gate turns that risk into a build failure you can see on a Tuesday afternoon instead of an outage on a Saturday night.

Limitations and who should skip this

This workflow needs request-level logs with token counts, and if your observability pipeline only stores aggregates you should fix that first rather than reconstructing a corpus from memory. It also assumes a relatively stable traffic shape; a product launch, a seasonal spike, or a new batch job invalidates the projection and requires a fresh corpus. The projection is an estimate with an explicit confidence boundary, not a guarantee, and you should re-run it monthly while the workload is young.

Skip this approach if you carry hard latency or availability obligations that a shared free endpoint cannot contractually meet, or if your data cannot legally leave its current processor. Skip it as well if you have no way to remove the old credential, because a migration you cannot reverse is a migration you should not start on a Friday. None of the code here makes a free tier suitable for regulated workloads; it only tells you whether the arithmetic works.

If you run the gate on your own corpus this week, the useful output is not the pass or fail line but the p95 request size, because that number tells you which single prompt in your product is quietly eating the budget. Start there, and let the cutover wait until the projection agrees with you.