Learn Prompt Budgets by Building a Tiny LLM Request Linter

Expected output first:

PASS short-answer: est_tokens=41 budget=120 canary=ok
FAIL oversized-context: est_tokens=173 budget=120 reason=budget_exceeded
FAIL missing-canary: est_tokens=58 budget=120 reason=canary_missing
SCHEMA-ERROR bad-role: content must be a string

The learning question is small: before a prompt reaches an LLM API, can we catch requests that are too large, malformed, or missing the one phrase we asked the model to repeat? I built this as a student-friendly way to practice prompt budgeting without paying for every experiment.

Prerequisites

• Python 3.11.x; I used only the standard library.
• No package install is required for the offline tests.
• Optional live check: any OpenAI-compatible chat endpoint. In class demos I sometimes point the optional live path at MonkeyCode because it offers free model access and a free server option, which is useful when the lesson is iteration rather than benchmark accuracy. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Create two files in an empty folder: prompt_budget.py and fixtures.json.

Complete runnable code

prompt_budget.py:

#!/usr/bin/env python3
import argparse, json, math, os, sys, urllib.request

ALLOWED_ROLES = {"system", "user", "assistant"}

def est_tokens(text: str) -> int:
    # Deliberately rough heuristic: about 4 chars/token for English-like text.
    # Use it only to teach budgets, not to invoice anyone.
    return max(1, math.ceil(len(text) / 4))

def load_messages(path):
    with open(path, "r", encoding="utf-8") as f:
        data = json.load(f)
    if not isinstance(data, list):
        raise ValueError("top level must be a list of messages")
    total = 0
    for i, m in enumerate(data):
        if not isinstance(m, dict):
            raise ValueError(f"message {i} must be an object")
        role = m.get("role")
        content = m.get("content")
        if role not in ALLOWED_ROLES:
            raise ValueError(f"message {i} has bad role: {role!r}")
        if not isinstance(content, str):
            raise ValueError("content must be a string")
        total += est_tokens(content)
    return total

def check(name, path, budget, canary):
    try:
        total = load_messages(path)
    except Exception as e:
        return f"SCHEMA-ERROR {name}: {e}"
    joined = "\n".join(m["content"] for m in json.load(open(path, encoding="utf-8")))
    if total > budget:
        return f"FAIL {name}: est_tokens={total} budget={budget} reason=budget_exceeded"
    if canary and canary not in joined:
        return f"FAIL {name}: est_tokens={total} budget={budget} reason=canary_missing"
    return f"PASS {name}: est_tokens={total} budget={budget} canary=ok"

def live(base_url, api_key, model, messages):
    req = urllib.request.Request(
        base_url.rstrip("/") + "/v1/chat/completions",
        data=json.dumps({"model": model, "messages": messages}).encode(),
        headers={"Authorization": "Bearer " + api_key, "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.loads(r.read().decode())["choices"][0]["message"]["content"]

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--budget", type=int, default=120)
    p.add_argument("--canary", default="BUDGET_OK")
    p.add_argument("--live", action="store_true")
    args = p.parse_args()
    for name, path in json.load(open("fixtures.json", encoding="utf-8")):
        print(check(name, path, args.budget, args.canary))
    if args.live:
        need = ["LLM_BASE_URL", "LLM_API_KEY", "LLM_MODEL"]
        missing = [k for k in need if not os.getenv(k)]
        if missing:
            sys.exit("Set " + ", ".join(missing) + " for --live")
        msgs = [{"role": "user", "content": "Reply with exactly: BUDGET_OK"}]
        print("LIVE:", live(os.getenv("LLM_BASE_URL"), os.getenv("LLM_API_KEY"), os.getenv("LLM_MODEL"), msgs))

fixtures.json:

[
  ["short-answer", "case_ok.json"],
  ["oversized-context", "case_big.json"],
  ["missing-canary", "case_canary.json"],
  ["bad-role", "case_bad.json"]
]

case_ok.json:

[
  {"role": "system", "content": "You are terse. Include BUDGET_OK."},
  {"role": "user", "content": "Explain tokens in one sentence."}
]

case_big.json: one user message whose content is 700 repeated characters, for example "x" * 700 written out as a long string. case_canary.json is like case_ok.json but omit BUDGET_OK. case_bad.json uses {"role": "user", "content": 42}.

Run:

python3 --version  # expect Python 3.11.x
python3 prompt_budget.py --budget 120 --canary BUDGET_OK

You should see the output block from the top. The optional live mode is intentionally separate:

LLM_BASE_URL=https://example.invalid LLM_API_KEY=replace-me LLM_MODEL=replace-me python3 prompt_budget.py --live

I would not run --live until every fixture is green. The linter is the lesson; the network call is only a smoke test.

What you should understand after finishing

A prompt budget is an application-level constraint, not a model guarantee. My estimator uses a rough four-characters-per-token heuristic because it keeps the project dependency-free; real tokenization differs by model and language, so validate against the tokenizer or API metadata before trusting a number. The useful habit is ordering: validate schema first, estimate cost second, check required strings third, and only then spend a network request. For current API shape, compare against the official OpenAI-compatible chat completions documentation you are using, and for tokenizer behavior read the Hugging Face tokenizers docs rather than assuming all models split text the same way.

Common mistakes

• Treating estimated tokens as exact billing tokens.
• Checking only total characters while ignoring message schema.
• Putting secrets in fixtures instead of environment variables.
• Testing only happy prompts; case_bad.json is where the validator earns its keep.

Limitations and who should not use this

This is not a production gateway, a cost calculator, or a model benchmark. Do not use it for regulated data, high-volume traffic, latency promises, or to compare providers. If you need exact token counts, use the model/provider tokenizer and pricing page. If your goal is prompt injection defense, this linter is insufficient; it only teaches one narrow budget habit.

Extension exercise: add a fixture where the canary appears in the system message but the user asks the model not to repeat it. Predict whether my joined-string check passes, then explain why a real evaluator should inspect the model output instead of the prompt. If you build that counterexample, share the smallest fixture that fools the linter.