Count Tokens per Span, Not per Run

A run total is an invoice. It tells you what you owe, and nothing about what to change. The moment an agent run costs more than you expected, the useful question is which span spent the tokens: the retrieval call, the tool schema you re-sent twelve times, the retry loop, the summarizer that quietly grew. One integer per run cannot answer that. A dashboard that refreshes hourly answers it after you have already burned the budget.

So the artifact here is small and boring. An append-only ledger records usage per span. A reconciler compares your local estimate against the provider's declared usage. A breaker refuses the next call once the run has crossed its cap. Standard library only, no tracing vendor, and no rewritten lines — which matters more than it sounds, because a crash at step nine of a twenty-step run should not cost you the first eight records.

This week's DEV front page swung between AI fatigue and open-source AI project pushes. Both are symptoms of the same gap: the tooling got loud, the accounting stayed quiet. Token spend is where that gap is cheapest to close.

The number you can't act on

Run-level totals conflate three different costs. There is the fixed cost of your prompt scaffolding, which you pay on every call whether or not the model needs it. There is the variable cost of tool output, which is where long JSON blobs and file reads land. And there is the cost of repetition: retries, re-plans, and the third attempt at the same failing edit.

A single total blends all three into a number that changes when any of them changes. You end up optimizing the scaffolding because it is the part you can see in your source, while the retry loop quietly doubles the bill. The ledger below exists so you can check that hypothesis against your own traffic instead of taking anyone's word for it — mine included.

A ledger that answers in spans

Write the record before you need it. Each span appends one open line and one closed line; nothing is ever mutated in place, so a killed process leaves a parseable prefix rather than a corrupt file.

# spend.py — append-only per-span token ledger with a budget breaker
import json, pathlib, time, uuid


class BudgetExceeded(RuntimeError):
    """Raised after the overshooting span is recorded, before the next call."""


class SpanLedger:
    def __init__(self, path="runs/ledger.jsonl", cap_tokens=200_000, reserve_tokens=2_000):
        self.path = pathlib.Path(path)
        self.path.parent.mkdir(parents=True, exist_ok=True)
        self.cap = cap_tokens
        self.reserve = reserve_tokens
        self.spent = 0
        self.run_id = uuid.uuid4().hex[:12]

    def _append(self, rec):
        line = json.dumps(rec, ensure_ascii=False, sort_keys=True)
        with self.path.open("a", encoding="utf-8") as f:
            f.write(line + "\n")

    def open(self, name, parent=None):
        self._append({"run_id": self.run_id, "span": name, "parent": parent,
                      "started_at": round(time.time(), 3), "status": "open"})
        return name

    def close(self, name, parent=None, usage=None, estimate_total=None):
        prompt = int((usage or {}).get("prompt_tokens", 0))
        completion = int((usage or {}).get("completion_tokens", 0))
        total = prompt + completion
        self.spent += total
        rec = {"run_id": self.run_id, "span": name, "parent": parent,
               "status": "closed", "prompt_tokens": prompt,
               "completion_tokens": completion, "total_tokens": total,
               "estimate_total": estimate_total,
               "drift": None if estimate_total is None else total - estimate_total,
               "cumulative_tokens": self.spent,
               "ended_at": round(time.time(), 3)}
        self._append(rec)
        return rec

    def guard(self):
        if self.spent + self.reserve > self.cap:
            raise BudgetExceeded(
                f"spent={self.spent} reserve={self.reserve} cap={self.cap}")

The ordering in close() is the whole design. The record is durably appended first, then guard() is called separately by the driver. That way the span which pushed you over budget is still in the file — you want to see it — and the refusal applies to the next call, which has not happened yet.

The reserve is not decoration. A breaker can only act on usage already reported, so it cannot stop the call in flight. You have to reserve headroom at least as large as your largest expected completion, otherwise the cap is a target rather than a limit.

ledger = SpanLedger(cap_tokens=120_000, reserve_tokens=4_000)
try:
    for step, prompt in enumerate(workload):
        span = f"step[{step}]"
        ledger.open(span, parent="run")
        reply = provider(prompt)              # must expose a usage payload
        ledger.close(span, parent="run", usage=reply.get("usage"),
                     estimate_total=estimate(prompt, reply.get("text", "")))
        ledger.guard()
        apply(reply)
except BudgetExceeded as e:
    log.warning("run halted: %s", e)

Reconcile the estimate against the declared usage

Every local tokenizer is wrong about someone's traffic. Tool schemas, minified JSON, and non-English text are the usual disagreements, because your estimator and the provider's tokenizer segment them differently. Store both numbers per span and the disagreement becomes a measurement rather than a mystery.

def reconcile(path, tolerance=0.05, top=5):
    rows = [json.loads(l) for l in pathlib.Path(path).read_text().splitlines() if l.strip()]
    closed = [r for r in rows if r.get("status") == "closed" and r.get("total_tokens")]
    flagged, abs_drift = [], 0
    for r in closed:
        est = r.get("estimate_total")
        if not est:
            continue
        rel = (r["total_tokens"] - est) / est
        abs_drift += abs(r["total_tokens"] - est)
        if abs(rel) > tolerance:
            flagged.append((r["span"], r["total_tokens"], est, round(rel, 3)))
    return {"spans": len(closed), "abs_drift_tokens": abs_drift,
            "flagged": sorted(flagged, key=lambda t: -abs(t[3]))[:top]}

Run it as python -m spend reconcile runs/ledger.jsonl --tolerance 0.05. The output is not a score. It is a list of spans whose cost you currently cannot predict, which is exactly the list to look at before you tune anything.

A three-check test plan you can run offline

The self-test below needs no network and no API key, because the provider is a stub. Treat the assertions as a specification of the behavior you want; expect to adjust the token arithmetic once you see your real provider's usage payload.

# selftest.py
import json, pathlib, tempfile, spend

calls = {"n": 0}

def fake_provider(prompt, completion_tokens=40):
    calls["n"] += 1
    return {"text": "x" * 10,
            "usage": {"prompt_tokens": len(prompt.split()),
                      "completion_tokens": completion_tokens}}

def estimate(prompt, text):
    return len(prompt.split()) + max(1, len(text) // 4)   # deliberately naive

P = "a b c d e f g h"
tmp = pathlib.Path(tempfile.mkdtemp())
ledger = spend.SpanLedger(tmp / "l.jsonl", cap_tokens=100, reserve_tokens=0)

ran = 0
try:
    for i in range(50):
        ledger.open(f"s{i}")
        r = fake_provider(P)
        ledger.close(f"s{i}", usage=r["usage"], estimate_total=estimate(P, r["text"]))
        ledger.guard()
        ran += 1
except spend.BudgetExceeded:
    pass

rows = [json.loads(l) for l in (tmp / "l.jsonl").read_text().splitlines()]
assert calls["n"] == ran + 1            # overshoot recorded, next call refused
assert sum(r.get("total_tokens", 0) for r in rows) <= 200
assert all(r["status"] in {"open", "closed"} for r in rows)

Three properties are worth checking before you trust the harness on real traffic. Does the breaker stop the next call rather than the current one? Does the sum of recorded spans match what you would have been billed? And does a SIGKILL mid-run leave a file you can still sum? The third one you can verify by hand in a minute: kill the process, then wc -l runs/ledger.jsonl and parse whatever is there.

Gate	Pass condition	What failure means
Breaker ordering	Overshooting span present, next call absent	You are capping after the fact
Reconciliation	Flagged spans are a short list	Estimator is too coarse to tune against
Crash safety	Truncated tail parses line-by-line	Something rewrites instead of appends

Where a free hosted run fits

None of this requires infrastructure — a JSONL file and a loop. What it does require is repetition: you need several comparable runs before the drift list means anything, and that is the point where local inference or a paid key starts to cost real money for what is essentially a measurement exercise.

MonkeyCode, an open-source coding agent project, is relevant here for a narrow reason: the operator states that it offers free model access and a free server option, with a free allowance reported as ten million tokens. I have not independently verified those terms, and allowances of this kind change. If you use that route, run the ledger harness against two or three runs and let the numbers decide whether it fits your workload; check the project's current terms before you plan around any figure. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limits, and who should skip this

The ledger is per-provider by construction. Token counts are tokenizer-specific, so summing spans across two providers gives you a number with no unit. Streamed responses may report usage late or not at all, in which case the span lands with total_tokens: 0 and your reconciliation silently skips it — worth an explicit assertion in production.

Skip this if you already reconcile a provider-side cost dashboard against invoices, or if you need per-tenant chargeback with currency and rounding rules; that is billing, and billing wants a real telemetry pipeline rather than a file you append to. Skip it too if your runs are single-call scripts — the run total and the span total are the same number, and you have gained a class and a file handle for nothing.

For everyone else the discipline is the deliverable. Record the span before you tune the prompt, reconcile the estimate before you trust it, and refuse the next call while refusal is still cheap.