Free tiers invite a specific kind of optimism: the first hundred calls cost nothing, so nobody measures them. Then a test suite gets pointed at the endpoint, the allowance disappears, and the team cannot say which calls ate it. The pairing hour described below is about closing that gap before it opens.
The session is a composite of the review a senior engineer typically runs on a "we tried the free tier" pull request. It is not a transcript, and the figures are placeholders. The artifacts are labelled: runnable in shape, unexecuted as printed.
The question that started the hour
The pull request added a model call inside a CI job. It worked. The description said the free tier covered it.
The senior asked one question and then stopped talking: what number is that claim based on?
Not "which model", not "how fast". Just the number. The author had a dashboard total, a rough per-request guess, and a green CI run. None of those three is a measurement of the same thing.
Dead end 1: the dashboard aggregate
The first instinct was to screenshot the provider console and divide by the number of test runs. That failed for a boring reason: the console total included the fixture-writing conversations from three days earlier, mixed with suite runs and a few manual probes.
An aggregate answers "how much did this account spend". It does not answer "how much does one pull request cost". Those are different denominators, and CI needs the second one.
- The console total could not be split by job.
- The job could not be replayed against the same counter.
- Nothing in the repository recorded which calls happened at all.
Dead end. The pair moved on.
Dead end 2: estimating with len(text) / 4
The second attempt was arithmetic. Character count divided by four, summed over the prompt, multiplied by a fudge factor for the completion.
It survived exactly one experiment. The suite sends JSON fixtures, indented, with long keys repeated across cases. Tokenizers do not treat whitespace and braces the way a character heuristic assumes, and the estimate drifted by a margin wide enough to flip the decision in both directions on different days.
An estimate that is sometimes high and sometimes low is not a conservative estimate. It is noise with a decimal point.
Dead end 3: wall-clock time as a cost proxy
Third attempt: if a run is slow, it must be expensive. Tempting, and wrong for the same reason cold-start timing is misleading in any free-tier environment — variance dominates. A queue hiccup on one run outweighed the real token delta across ten others.
Latency is worth measuring. It just measures a different thing than usage.
The artifact we kept: a usage ledger and a fail-closed gate
Two small files, no framework. The first wraps every call and appends one JSON line per request.
# budget_probe.py — illustrative, not executed as printed.
# Appends one ledger row per call to an OpenAI-compatible endpoint.
import json, os, time, urllib.request
BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
LEDGER = os.environ.get("BUDGET_LEDGER", "usage.jsonl")
def call(prompt: str, tag: str) -> dict:
body = json.dumps({
"model": os.environ["LLM_MODEL"],
"messages": [{"role": "user", "content": prompt}],
}).encode()
req = urllib.request.Request(
f"{BASE}/chat/completions",
data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {KEY}"},
)
started = time.monotonic()
with urllib.request.urlopen(req, timeout=120) as resp:
payload = json.load(resp)
usage = payload.get("usage") or {}
row = {
"ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"tag": tag,
"model": payload.get("model"),
"prompt_tokens": usage.get("prompt_tokens"),
"completion_tokens": usage.get("completion_tokens"),
"seconds": round(time.monotonic() - started, 3),
}
with open(LEDGER, "a", encoding="utf-8") as fh:
fh.write(json.dumps(row) + "\n")
return row
The second file reads the ledger and refuses to guess. This is the part that mattered in review: unmeasured calls fail the gate instead of being silently averaged into a comfortable number.
# budget_gate.py — illustrative, not executed as printed.
import json, sys
from datetime import datetime
ALLOWANCE = int(sys.argv[1]) # operator-declared allowance, in tokens
DAYS = int(sys.argv[2]) if len(sys.argv) > 2 else 30
rows, missing = [], 0
for line in open("usage.jsonl", encoding="utf-8"):
r = json.loads(line)
if r["prompt_tokens"] is None or r["completion_tokens"] is None:
missing += 1
continue
rows.append(r)
if not rows:
sys.exit("no measured rows: refusing to project")
used = sum(r["prompt_tokens"] + r["completion_tokens"] for r in rows)
stamps = sorted(datetime.fromisoformat(r["ts"].replace("Z", "+00:00")) for r in rows)
span = max((stamps[-1] - stamps[0]).total_seconds() / 86400, 1 / 24)
per_day = used / span
projected = per_day * DAYS
print(f"rows={len(rows)} unmeasured={missing} used={used} per_day={per_day:.0f}")
print(f"projected={projected:.0f} allowance={ALLOWANCE} runway_days={ALLOWANCE / per_day:.1f}")
if missing:
sys.exit("fail closed: unmeasured calls make the projection a guess")
if projected > ALLOWANCE * 0.8:
sys.exit("headroom under 20%: do not send the full suite through this tier")
Run it as python budget_gate.py <allowance> 30. The threshold is deliberately blunt: eighty percent, not ninety-five, because the suite grows and the allowance does not.
The canary that keeps the ledger honest
Accounting drifts when the provider changes how it counts, or when a proxy strips the usage block. A fixed canary prompt — same text every week, stored in the repository — gives a stable token count to compare against. A shift of more than a few tokens is a signal to re-read the terms, not a signal to edit the threshold.
The decision table the pair wrote down
| Signal from the ledger | Action |
|---|---|
| Usage is measured, headroom above 20% | Keep the free tier for CI |
| Usage is measured, headroom below 20% | Reduce suite to a labelled subset; keep the full suite manual |
unmeasured > 0 |
Stop. Fix instrumentation before scaling calls |
| Canary token count drifts week over week | Re-verify allowance terms and tokenizer behaviour |
| Per-day burn exceeds what any free tier plausibly covers | Move the heavy path to a paid or self-hosted endpoint |
Where a free tier and a free server actually fit
The operator describes free model access and a free server option for this workflow. That combination is genuinely useful for the first week of a ledger: the numbers you are collecting are exactly the numbers that tell you whether the free arrangement is enough.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Two cautions before pointing a suite at it. Allowances and availability change, so read the current terms rather than a post from any month, including this one. And a free server is still a server: measure its cold-start behaviour separately, because latency variance will contaminate any cost reasoning you try to do from timing alone.
Limitations and who should skip this
- The ledger assumes the endpoint returns a
usageblock. If it does not, the gate fails closed by design and the approach stalls until you measure server-side. - Token counts are not currency. If your provider bills in units you cannot map to tokens, this table is the wrong abstraction.
- Teams under a hard data-residency rule should not route fixtures through a hosted free tier at all, regardless of allowance.
- Anyone who needs a guaranteed latency floor should skip the free-server path and pay for isolation.
The boring conclusion held up in review: the ledger was worth more than the model. If the free access disappears tomorrow, the gate still tells you what the suite costs and where to cut it. That is the part you keep.
If you want to try the shape of this, start with one job, one canary prompt, and one week of rows before you trust any allowance number — including the one in this article.
Top comments (0)