A Token Ledger Decides Which Model Answers

The failure is boring and repeatable. A script fans out one request per changed file, the longest file drags the whole prompt past what the cheap endpoint will accept, the retry loop fires anyway, and a code review turns into a spend event. Nobody chose that. It fell out of a for loop that never asked how big the thing it was sending actually was.

So for two days I stopped tuning prompts and instrumented the decision instead. The question shifted from which model is better to which call is allowed to happen at all, and that turned out to be answerable with about sixty lines of Python and a JSONL file.

The ledger comes before the router

Most routing advice starts with model quality. That ordering is backwards for a side project, because quality is the thing you can't measure cheaply and volume is the thing you can. Write down every call first. Then argue about where it goes.

The estimator below is crude on purpose. It is deterministic, it never needs a network hop, and it is pessimistic about code, which is where my prompts actually live.

# ledger.py — record the call before you spend anything
import hashlib, json, os, time

LEDGER = os.environ.get("LEDGER_PATH", "ledger.jsonl")
# deliberately pessimistic: ~3.6 chars per token for mixed prose + code
CHARS_PER_TOKEN = 3.6

def estimate_tokens(text: str) -> int:
    return int(len(text) / CHARS_PER_TOKEN) + 1

def route(prompt: str, needs_schema: bool) -> str:
    if needs_schema:
        return "pinned"          # strict JSON output goes where the model is fixed
    if estimate_tokens(prompt) > 1200:
        return "pinned"          # too big for the free path in my setup
    return "free"

def record(job_id: str, prompt: str, choice: str, started: float) -> None:
    row = {
        "job_id": job_id,
        "chars": len(prompt),
        "est_tokens": estimate_tokens(prompt),
        "route": choice,
        "wall_s": round(time.time() - started, 3),
        "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()[:12],
    }
    with open(LEDGER, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")

Two things matter here. The hash lets you find duplicate prompts without storing the prompt itself, which I wanted because review diffs are not mine to keep. And wall_s is measured, not estimated, so the free path can be judged on latency with real numbers instead of vibes.

The route table is the whole design

There is no cleverness in the routing rule. It is a lookup on two booleans and a size, and the value is that the rule is written down where a reviewer can disagree with it.

Condition	Route	Why
Prompt needs strict JSON or a pinned model version	pinned	schema drift is worse than cost
Estimated tokens over the free-path ceiling	pinned	truncation mid-diff produces silent garbage
Anything else	free	this is the volume case, and volume is where free access matters
Retry after a failure	pinned	retries are not experiments, pay for determinism

That last row is the one I'd defend hardest. A retry with a different sampling outcome is not a retry, it's a second experiment, and experiments should be labelled as such.

Two days, one summary script

Instrumentation you never read is just disk usage. I ran a summarizer every evening, and the shape of the output is the artifact worth copying:

# summarize.py — per-route distribution, no dashboard required
import json, statistics, collections

rows = [json.loads(l) for l in open("ledger.jsonl", encoding="utf-8")]
by_route = collections.defaultdict(list)
for r in rows:
    by_route[r["route"]].append(r)

for route, items in sorted(by_route.items()):
    toks = sorted(i["est_tokens"] for i in items)
    walls = sorted(i["wall_s"] for i in items)
    p95 = toks[max(0, int(len(toks) * 0.95) - 1)]
    print(f"{route:7s} n={len(items):4d} "
          f"tokens_median={statistics.median(toks):6.0f} p95={p95:6d} "
          f"wall_median={statistics.median(walls):.2f}s")

seen = collections.Counter(r["prompt_sha"] for r in rows)
dupes = sum(c - 1 for c in seen.values() if c > 1)
print(f"duplicate prompts: {dupes} of {len(rows)}")

The duplicate count was the number I did not expect to care about. Repeated prompts are pure waste on any tier, free or not, and they are invisible until you hash for them.

I am not going to publish a savings percentage from my own two-day window, because a single repo over a single weekend is not evidence and quoting it would be the same mistake as quoting a benchmark nobody can rerun. The script is the claim. Run it against your own traffic and the numbers arrive in an evening.

Running the router on a small free box

The router does almost nothing — one file, one port, no database — which makes it a fair candidate for a small instance rather than a paid one. MonkeyCode's free model access and free server option are the two pieces this workflow leans on: the free path serves the volume case, and the free instance hosts the ledger service.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I treat both as operator-supplied availability claims and verify them in my own environment before trusting either. That means a smoke test on day zero, not day thirty:

# day 0: does the small box survive a cold start and a burst?
for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
    -X POST http://127.0.0.1:8080/route \
    -H 'content-type: application/json' \
    -d "{\"prompt\":\"review file $i\",\"needs_schema\":false}"
done | sort | uniq -c | sort -rn

A systemd unit keeps it honest, and Restart=on-failure is the only resilience claim I make:

[Unit]
Description=model-router
After=network-online.target

[Service]
ExecStart=/usr/bin/python3 /opt/router/app.py
Environment=LEDGER_PATH=/var/lib/router/ledger.jsonl
Restart=on-failure

[Install]
WantedBy=multi-user.target

Then journalctl -u model-router -f while the burst runs. If the free instance sleeps, you will see it in wall_s long before you see it in a bill.

What broke, and what I'd repeat

What broke was predictable: the estimator over-counted long diffs badly enough to push mid-size prompts onto the pinned path, and my first ceiling was set too low. Raising it by eye is not a fix — the fix is reading tokens_median and p95 for the free route and setting the ceiling at the 95th percentile of what you actually send.

What I'd repeat is the ordering. Ledger, then route, then deploy. The reverse order gives you a routing policy you can't audit and a server you're afraid to restart.

Two limits worth stating plainly. Token estimates from character counts are wrong by a wide margin on minified files and on non-Latin text, so don't budget on them; use them only to compare calls against each other. And this design says nothing about answer quality — it decides where a call goes, not whether the answer was good.

Who should skip this: anyone with a latency SLA, anyone whose diffs cannot leave their network, and anyone whose traffic is small enough that the ledger costs more attention than the calls cost money. If you are any of those, a hard-coded single endpoint is the correct engineering choice.

If you do want to try the split without a card on file, the free model access and the free server are where I'd start — and bring the ledger with you, not the trust.