Your Token Budget Is a Test, Not a Dashboard

Most teams learn about prompt bloat from an invoice. That is the wrong feedback loop. A token budget should fail a pull request, not a finance review.

My position is blunt. Cost assertions belong in version control, next to the prompt they measure. Dashboards are for exploration. Tests are for enforcement.

Dashboards are a lagging signal

Observability tells you what already shipped. It aggregates across branches, so nobody owns the number. By the time the chart bends, the regression is already on main.

A gate behaves differently. It blocks the merge, names the fixture, prints the delta. One reviewer decides: fix the prompt, or accept the growth on purpose.

What a budget gate asserts

1. Assembling fixture x costs no more than baseline plus tolerance.
2. A new fixture arrives as a reviewed commit, not a side effect.
3. A tokenizer or model change bumps a counter_version field.

Point three matters most. Token counts are not portable across tokenizers. A silent tokenizer upgrade can swing every number at once. Version the counter, and your baseline diff stays honest.

Fixtures, not live traffic

Never call a provider inside the blocking job. Store inputs as files. Record counts in a separate refresh job you can rerun at will.

Suggested layout

budget/
  fixtures/summarise.txt
  fixtures/classify.txt
  baseline.json
  gate.py
  record.py
  test_gate.py

baseline.json is small, reviewable data:

{
  "counter_version": "word-proxy-1",
  "fixtures": {
    "summarise": 120,
    "classify": 40
  }
}

The gate: sixty lines, no network

The gate compares numbers. It never touches a model.

# budget/gate.py
"""Fail a build when prompt assembly grows past a recorded baseline."""
import json
import sys
from pathlib import Path

TOL = 0.05  # allowed growth per fixture


def load(path: str) -> dict:
    return json.loads(Path(path).read_text(encoding="utf-8"))


def compare(baseline: dict, current: dict, tol: float = TOL):
    rows, failures = [], []
    for name, base in sorted(baseline["fixtures"].items()):
        cur = current["fixtures"].get(name)
        if cur is None:
            failures.append((name, "fixture missing from current run"))
            continue
        growth = (cur - base) / base if base else float("inf")
        rows.append((name, base, cur, growth))
        if growth > tol:
            failures.append((name, f"{growth:+.1%} over baseline of {base}"))
    return rows, failures


def main(argv: list[str]) -> int:
    baseline, current = load(argv[1]), load(argv[2])
    tol = float(argv[3]) if len(argv) > 3 else TOL
    if baseline["counter_version"] != current["counter_version"]:
        print("counter_version changed: re-record the baseline in its own commit")
        return 2
    rows, failures = compare(baseline, current, tol)
    for name, base, cur, growth in rows:
        print(f"{name:<24} {base:>7} -> {cur:>7}  {growth:+.1%}")
    if failures:
        print("\nbudget gate failed:")
        for name, why in failures:
            print(f"  - {name}: {why}")
        return 1
    print("\nbudget gate passed")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Run it locally against two JSON files. That is the whole contract.

python budget/gate.py budget/baseline.json budget/current.json 0.05

Prove the gate before you trust it

A gate nobody tested is a gate everyone disables. Keep three offline tests.

# budget/test_gate.py
import unittest
from budget.gate import compare

BASE = {
    "counter_version": "word-proxy-1",
    "fixtures": {"summarise": 120, "classify": 40},
}


def current(summarise: int, classify: int = 40) -> dict:
    return {
        "counter_version": "word-proxy-1",
        "fixtures": {"summarise": summarise, "classify": classify},
    }


class GateTest(unittest.TestCase):
    def test_small_growth_passes(self):
        rows, failures = compare(BASE, current(124), tol=0.05)
        self.assertEqual(failures, [])

    def test_bloat_fails(self):
        rows, failures = compare(BASE, current(180), tol=0.05)
        self.assertEqual(len(failures), 1)

    def test_missing_fixture_fails(self):
        data = current(120)
        data["fixtures"].pop("summarise")
        rows, failures = compare(BASE, data, tol=0.05)
        self.assertEqual(len(failures), 1)


if __name__ == "__main__":
    unittest.main()

Run them with no network and no credentials.

python -m unittest budget.test_gate -v

If these pass, the logic is sound. Any later failure is a real prompt change, not a bug in your harness.

Count with a proxy first, a real tokenizer second

Start with something deterministic and obviously fake.

# budget/count.py
class WordProxy:
    """A stand-in, not a tokenizer. Use it to prove the plumbing works."""
    version = "word-proxy-1"

    def count(self, text: str) -> int:
        return len(text.split())

WordProxy counts words. It is a placeholder, not a billing estimate. Ship it only to validate the pipeline.

Then swap in your provider's tokenizer for real numbers. Bump counter_version in the same commit. Otherwise every fixture looks like a regression at once.

Wire it into CI with a disposable runner

Keep the blocking job cheap and offline.

name: budget-gate
on: [pull_request]
jobs:
  budget:
    runs-on: [self-hosted, budget-runner]
    steps:
      - uses: actions/checkout@v4
      - run: python -m budget.record --offline --out budget/current.json
      - run: python budget/gate.py budget/baseline.json budget/current.json 0.05

A self-hosted runner fits this job well. Re-recording is bursty and boring, and it should never touch your laptop.

This is where a free tier is genuinely useful. MonkeyCode's operator describes free model access and a free server option, with an advertised allowance of 10 million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not benchmarked it, and I am not claiming it replaces a paid quota.

Treat that machine as a runner, not a pet. If it disappears, record.py reruns somewhere else. Nothing in the gate depends on it.

The baseline commit is the weak link

Every cost gate dies the same way. Someone refreshes the baseline to unblock a merge.

Four rules keep it alive:

• Baseline edits need a reviewer who is not the author.
• Require a one-line reason in the commit body.
• Never refresh the baseline inside the gate job.
• Pin one counter_version per commit.

Without these, just refresh the baseline becomes the new just rerun the flaky test.

When the gate fails, debug in this order

1. Read the failing fixture name; ignore the rest.
2. Diff the prompt template for that fixture only.
3. Check injected context: retrieval chunks, tool schemas, few-shot examples.
4. Confirm the counter version before editing any prompt.
5. Fix the smallest cause. Rerun the gate locally.
6. If the growth is intentional, bump the baseline in its own commit.

Most failures land on step three. Someone adds a document chunk or a tool schema. Cost climbs and quality does not.

Where this approach breaks

• Retrieval fixtures are nondeterministic; snapshot the retrieved chunks instead of the query.
• Prompt caching and reasoning tokens can change billed cost without changing prompt text.
• Under five prompts, the bookkeeping costs more than it saves.
• Prompts edited in a vendor dashboard are invisible to git; this gate cannot see them.
• Tokenizers move under you; record the baseline deliberately, never automatically.

Decision table

Situation	Gate in CI?	Reason
5-20 stable prompts in the repo	Yes	Cheap to maintain, catches drift early
Prompts edited in a vendor UI	No, export first	The gate cannot read what git cannot see
Heavy RAG pipeline	Yes, with chunk snapshots	Retrieval is the usual source of bloat
Multi-tenant prompt variants	Yes, one fixture per variant	A single variant can regress alone
Throwaway prototype	No	Tolerance tuning is wasted effort
Cost driven by output length	Different test	Assert on max_tokens, not prompt size

Who should skip this

Skip it if your prompts are not files. Skip it if your spend is dominated by long completions. Skip it if nobody will review a baseline diff, because then the gate is decoration.

The takeaway

Cost regressions are code regressions. They deserve the same enforcement. Put the number in the repo, fail the pull request, and let a human decide when the number is allowed to move.

If you need a throwaway runner for the refresh job, MonkeyCode's free server option is worth a look. The gate itself should never depend on it.