Case Study: Sizing Inputs Before the Call, Not After the Bill

The cheapest token is the one you never spend, so the most useful budget control sits before the request, not in the invoice you read on Monday. This case study walks through a small changelog bot that kept running out of quota mid-week, and the pre-flight sizing layer that turned a soft budget into a hard boundary. The artifact is about 120 lines of dependency-free Python you can drop into any worker that pays per token.

Background: one paragraph per merged pull request

The project was deliberately boring. A worker polled a repository every ten minutes, collected merged pull requests, and asked a model for a one-paragraph changelog entry that was posted to an internal channel. Volume was roughly thirty to forty pull requests a day, which sounds trivial until you look at the inputs. Many pull request threads contained long review discussions, quoted replies, and pasted stack traces, so the same item could be 400 tokens on a quiet day and 9,000 tokens on a noisy one.

The worker ran on free model access, which made throughput cheap but made quota the binding constraint rather than latency. When the daily allowance ran out, the worker did not stop cleanly. It hit context-length errors on the largest threads, retried with the same oversized input, and burned the remaining budget on requests that could never succeed.

Goal: three properties the fix had to have

1. The daily cap must be a boundary the process cannot cross, not a number you discover after the fact.
2. No pull request may be silently dropped when the budget runs out; it must be parked with a machine-readable reason.
3. Every decision must be reproducible from files on disk, so a restart costs nothing and a re-run tells the same story.

Notice what is absent from that list: accuracy improvements, model upgrades, prompt cleverness. The interesting failure was arithmetic, and arithmetic failures get fixed with arithmetic.

Step 1: normalize before you measure

Half the token bill in this bot was quoted text and bot chatter that never influenced the summary. Normalizing first is free, deterministic, and shrinks the estimation problem as well as the request.

import re

QUOTE = re.compile(r"^\s*>.*$", re.M)
COLLAPSE = re.compile(r"\n{3,}")
SPACES = re.compile(r"[ \t]{2,}")

def normalize(text: str, drop_markers=("<!--", "ci-bot:", "dependabot")) -> str:
    text = QUOTE.sub("", text)
    kept = [ln for ln in text.splitlines() if not any(m in ln for m in drop_markers)]
    text = SPACES.sub(" ", "\n".join(kept))
    return COLLAPSE.sub("\n\n", text).strip()

Run normalize on your own corpus for a week and log len(before) - len(after). If the reduction is under ten percent, your inputs are already clean and this step is noise; if it is above thirty percent, you were paying for quoted replies.

Step 2: estimate with two ratios, not one

A single characters-per-token ratio is wrong the moment your input contains code. Fenced blocks tokenize densely around punctuation and indentation, so the estimator below splits the text and applies a different divisor to each region, then adds a fixed overhead for message framing.

FENCE = re.compile(r"```

[\s\S]*?

```")
MESSAGE_OVERHEAD = 24

def estimate(text: str, factor: float = 1.0) -> int:
    code_chars = sum(len(m) for m in FENCE.findall(text))
    prose_chars = len(text) - code_chars
    raw = code_chars / 3.2 + prose_chars / 4.3 + MESSAGE_OVERHEAD
    return int(raw * factor)

The factor argument is the whole calibration story. After every real call you write the reported usage back into a small JSON file, recompute actual / estimated over the last fifty calls, and store the median as factor. You are not trying to be exact; you are trying to be consistently biased in one known direction.

Step 3: choose a plan from a table, not from a guess

Once you have an estimate, the routing decision is deterministic. The thresholds below are the ones this bot uses, and they are conservative on purpose because an underestimate costs a failed call while an overestimate costs a slightly worse summary.

Estimated tokens	Plan	Max model calls	Failure mode avoided
≤ 1,500	single	1	nothing, this is the happy path
1,501 – 4,000	trim	1	context-length errors on long threads
> 4,000	map_reduce	5	silent truncation of review discussion

def plan(text: str, limits=(1500, 4000), max_sections=4) -> tuple[str, list[str]]:
    est = estimate(text)
    if est <= limits[0]:
        return "single", [text]
    if est <= limits[1]:
        return "trim", [head_tail(text, head=1200, tail=600)]
    return "map_reduce", split_sections(text, max_sections)

head_tail keeps the first 1,200 characters and the last 600, because the opening message usually states the intent and the tail usually holds the resolution. split_sections cuts on markdown headings and falls back to equal chunks, hard-capped at four so the worst case is five calls per item.

Step 4: reserve the budget before the request leaves the process

The reservation happens in memory and on disk at the same time, and a rejected reservation is a normal outcome rather than an error. This is the difference between a budget you hope to respect and one you cannot violate.

import json, pathlib, datetime as dt

LEDGER = pathlib.Path("budget.json")

def load() -> dict:
    if LEDGER.exists():
        return json.loads(LEDGER.read_text())
    return {"day": "", "reserved": 0, "spent": 0, "deferred": []}

def reserve(state: dict, cost: int, daily_cap: int) -> bool:
    today = dt.datetime.now(dt.timezone.utc).date().isoformat()
    if state["day"] != today:
        state.update(day=today, reserved=0, spent=0)
    if state["reserved"] + cost > daily_cap:
        return False
    state["reserved"] += cost
    LEDGER.write_text(json.dumps(state, indent=2))
    return True

When reserve returns False, the worker appends the pull request to deferred with the estimate, the plan name, and the shortfall, then moves on. That list is the single most useful debugging artifact in the whole system, because it tells you whether you are under-budgeted or over-chunking.

Step 5: reconcile actuals against reservations

After each successful call, subtract the reservation and add the real usage, then refresh factor once a day. This keeps the ledger honest without requiring exact prediction, and it makes the weekday shape visible: this bot's Monday is heavier than its Thursday, which is obvious in hindsight and invisible in a monthly total.

Where MonkeyCode fits

This worker calls a model through MonkeyCode's free model access, and the operator states that the free tier includes a free allowance of ten million tokens plus a free server option for running the worker itself. Treat those two numbers as claims to verify against the current product page before you size anything, since tier details are exactly the kind of thing that drifts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option mattered less for compute and more for process hygiene: because the ledger lives in a single JSON file outside the process, a restart loses nothing and a redeploy re-reads the same state. That property is worth having regardless of where the worker runs.

The harness: three tests anyone can re-run

1. Estimator bias. Feed fifty recorded inputs through estimate and compare to the usage numbers your own calls reported; assert the median ratio sits between 0.8 and 1.25 and fail the build outside that band.
2. Planner monotonicity. Assert that estimate never decreases as you append text, and that a plan never produces more sections than max_sections.
3. Cap invariant. Drive reserve with a synthetic cost sequence and assert reserved never exceeds daily_cap, including across a simulated date rollover.

Each test runs offline against fixtures, so you can validate the budget layer without spending a token.

Results

I am not going to quote a savings percentage here, because the number is entirely a function of your corpus and would be theater. What the change does guarantee, by construction of the code above, is narrower and still worth stating: the process cannot exceed its daily reservation, oversized items are split into at most five calls instead of retried whole, and every skipped item leaves a row explaining itself.

Limitations and who should not use this

• If your inputs are uniformly small, the estimator is overhead; a fixed cap is enough.
• If your provider bills per request rather than per token, reservation arithmetic buys you little.
• If your summaries must be exhaustive rather than representative, head_tail trimming will lose information you care about.
• If your corpus shifts weekly, a static factor will drift and you should recalibrate on a schedule rather than once.

Closing note

If you want to try the shape, start with the estimator harness and fifty recorded inputs from your own worker, then add the planner only after you know your bias direction. The parts are boring on purpose, and boring is what you want in the component that decides whether the rest of the system gets to run.