Token Forensics for Agent Runs: Five Waste Signatures and the Order to Fix Them

A team I worked with had a nightly agent job that was supposed to run for six hours. By 11 a.m. its daily allowance was gone, and the run had produced two usable summaries out of nine.

Nobody could say which step ate the tokens. The router logs showed requests; they did not show cost. So the first instinct was to blame the model.

It was not the model. It was step 7 resending the same 4,000-token tool output on every attempt, after a validation failure that fired four times.

That is the whole problem in miniature: agent budgets rarely die from one big call. They die from structural waste that no request log records.

Below is the accounting I now run before changing anything — a small meter, five waste signatures, and the order in which to fix them.

1. Start with the report, not the router

Routing decisions are the wrong first lever. If 40% of your billed tokens are retries of a call that always fails, no amount of re-routing helps; you will just fail more cheaply.

Measure per step first. Then the routing question becomes answerable with numbers instead of vibes.

Glossary (six terms, used consistently below)

Term	Definition	Signal it produces
Billed tokens	Sum of prompt and completion tokens the provider counts for a run	Absolute cost
Static tokens	System prompt plus tool schemas, paid on every single call	overhead_ratio
Reused tokens	Context carried into a step unchanged from an earlier step	redundancy_ratio
Retry amplification	Share of billed tokens spent on attempts that ended in failure	retry_amplification
Prompt growth	How much the last successful prompt outgrew the first	prompt_growth
Leaf	The single branch the decision tree lands on, i.e. one prescribed first action	leaf

The sixth term matters most. A tree that returns three suggestions per run is a tree nobody follows.

2. The meter (stdlib only, ~60 lines)

You need three numbers per step that providers already return in usage, plus two annotations you add yourself: how many tokens are static, and how many are reused from an earlier step.

The annotations are manual or heuristic. That is a feature — it forces you to look at the prompt.

#!/usr/bin/env python3
"""meter.py - per-step token accounting for agent runs. Python 3.10+, stdlib only."""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass


@dataclass
class Step:
    name: str
    prompt_tokens: int
    completion_tokens: int
    static_tokens: int = 0   # system prompt + tool schemas, paid per call
    reused_tokens: int = 0   # context carried over unchanged from an earlier step
    attempt: int = 1
    ok: bool = True


def load(path: str) -> list[Step]:
    with open(path, encoding="utf-8") as fh:
        return [Step(**row) for row in json.load(fh)]


def summarize(steps: list[Step]) -> dict:
    prompt = sum(s.prompt_tokens for s in steps)
    completion = sum(s.completion_tokens for s in steps)
    if not steps or prompt == 0:
        return {"steps": len(steps), "billed_tokens": prompt + completion}

    billed = prompt + completion
    wasted_retry = sum(
        s.prompt_tokens + s.completion_tokens for s in steps if not s.ok
    )
    ok_steps = [s for s in steps if s.ok]
    first = ok_steps[0].prompt_tokens if ok_steps else 0
    last = ok_steps[-1].prompt_tokens if ok_steps else 0

    return {
        "steps": len(steps),
        "billed_tokens": billed,
        "overhead_ratio": round(sum(s.static_tokens for s in steps) / prompt, 3),
        "retry_amplification": round(wasted_retry / billed, 3),
        "redundancy_ratio": round(sum(s.reused_tokens for s in steps) / prompt, 3),
        "prompt_growth": round((last - first) / first, 3) if first else 0.0,
    }


ACTIONS = {
    "B_retry": "install a stop rule before touching anything else",
    "A_overhead": "shrink the system prompt and tool schemas",
    "C_redundancy": "collapse steps or externalize state",
    "D_drift": "cap the transcript, move history to a scratch file",
    "F_leave_it": "budget is healthy; change nothing",
}


def classify(m: dict) -> str:
    if not m.get("billed_tokens"):
        return "F_leave_it"
    if m["retry_amplification"] > 0.25:
        return "B_retry"
    if m["overhead_ratio"] > 0.60:
        return "A_overhead"
    if m["redundancy_ratio"] > 0.40 and m["steps"] >= 8:
        return "C_redundancy"
    if m["prompt_growth"] > 0.30:
        return "D_drift"
    return "F_leave_it"


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print("usage: meter.py steps.json", file=sys.stderr)
        return 2
    report = summarize(load(argv[1]))
    leaf = classify(report)
    report["leaf"] = leaf
    report["first_action"] = ACTIONS[leaf]
    print(json.dumps(report, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Input is a flat JSON array. Three rows are enough to show the shape:

[
  {"name": "plan",            "prompt_tokens": 3200, "completion_tokens": 410, "static_tokens": 2400, "reused_tokens": 0,    "attempt": 1, "ok": true},
  {"name": "fetch_schema",    "prompt_tokens": 5100, "completion_tokens": 120, "static_tokens": 2400, "reused_tokens": 2600, "attempt": 1, "ok": true},
  {"name": "validate_schema", "prompt_tokens": 5400, "completion_tokens": 90,  "static_tokens": 2400, "reused_tokens": 2600, "attempt": 3, "ok": false}
]

Run it:

python meter.py steps.json

Note: the script is dependency-free Python 3.10+. I have exercised it only against synthetic usage dicts, so treat its output as a structural description of your run, not as a benchmark of any model or provider.

3. The decision tree, in fix order

The branches are ordered by cost of ignorance, not by elegance. Retries are checked first because they are the cheapest thing to fix and the most expensive thing to leave alone.

1. retry_amplification > 0.25 → leaf B. A quarter of the budget is being spent on attempts that fail. Stop the run earlier.
2. overhead_ratio > 0.60 → leaf A. Most of each prompt is boilerplate you wrote once and now pay forever.
3. redundancy_ratio > 0.40 and at least 8 steps → leaf C. Long runs are resending the same material.
4. prompt_growth > 0.30 → leaf D. The transcript is growing faster than the work.
5. None of the above → leaf F. Do nothing. This leaf is real and should fire often.

The stop rule for leaf B

A ratio is a diagnosis; you need a hard stop in code.

DAILY_ALLOWANCE = 1_000_000          # your number, from your own account
MAX_WASTE_RATIO = 0.25
wasted = 0

for step in steps:
    result = run_step(step)
    if not result.ok:
        wasted += result.prompt_tokens + result.completion_tokens
    if wasted > DAILY_ALLOWANCE * MAX_WASTE_RATIO:
        raise SystemExit("stop: waste budget exhausted, fix the failing step first")

This turns an invisible leak into a visible crash. Crashes get fixed; leaks get re-billed.

4. Worked examples at each leaf

The numbers below are illustrative workloads, not measurements of any product.

Leaf B — retry amplification 0.41. Nine steps, 180k billed tokens, 74k of them on failed validation attempts. First action: make the validator return the exact field path, not "invalid input". Second: apply the stop rule above. Expected effect is bounded — you cap the loss, you do not recover the tokens.

Leaf A — overhead ratio 0.72. A 3,100-token system prompt plus four tool schemas against a 4,300-token average prompt. First action: move three rarely used tools behind a second call. If a schema is only needed after planning, it does not belong in the planner's prompt.

Leaf C — redundancy 0.55 with 12 steps. The same repository map is pasted into eight prompts. First action: externalize it to a file the agent reads by path, and pass a one-line index instead. Second: check whether steps 4 and 5 can merge — often they exist because of an earlier ordering constraint that no longer holds.

Leaf D — prompt growth 0.84. The first successful prompt was 2,900 tokens; the last was 5,300. First action: summarise every third turn into a scratch note and drop the raw turns. Second: measure again — if growth persists, the agent is accumulating state it never reads.

Leaf F — everything under threshold. The correct output of this workflow is often "no change". Record the report, set an alert on retry_amplification, and go do something else.

5. Where a free tier actually changes the math

One leaf benefits from cheap capacity, and it is not the one people expect.

Leaf C and leaf D often need a summariser: a step that compresses history before the expensive step reads it. That step is repetitive, low-stakes, and easy to validate — a good candidate for a free endpoint, because a bad summary fails loudly at the next step rather than silently.

That is where MonkeyCode fits in this workflow rather than in the headline. The project advertises free model access and a free server option, and an allowance of 10 million tokens at the time of writing — operator-stated figures that you should confirm against current terms before planning around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I would not move the planning step there first. Move the step whose failure is easiest to detect, run the meter again, and compare billed_tokens before and after. If the number does not move, the free tier was a distraction, not an optimisation.

6. Limitations, and who should skip this

• Usage accounting is not portable. Cached tokens, reasoning tokens, and tool-call tokens are reported differently across providers. Normalise or your ratios lie.
• The annotations are manual. static_tokens and reused_tokens come from your reading of the prompt, not from the API.
• It measures spend, not quality. A run can hit leaf F and still be useless. Pair this with a task-level success check.
• Skip it if you already have tracing. If your stack records per-span token counts and you trust them, use those numbers and borrow only the fix order.
• Skip it if latency dominates. On a latency-bound pipeline, shrinking prompts for token reasons can cost you more than it saves.

Closing

The order matters more than the thresholds. Retries, then boilerplate, then redundancy, then growth — and then, usually, nothing.

If you want to test this today, save one run's usage fields to JSON and run the meter against it. It is small enough to finish before lunch, and the leaf it returns is a better starting point than another router rewrite. If you decide the summariser step is worth moving, the project's free tier is a reasonable place to try it — with the meter still running.