The Token Leader Is Not the Latency Leader: Build a Span Ledger for Agent Runs

An agent runs overnight. In the morning the free tier is gone and the log says loop finished. Nobody knows which step spent the budget.

The trace already has the answer. It is just not sorted the way we need. Most trace viewers rank spans by time. Cost is a different axis, and it disagrees.

This post builds a small span ledger. It answers two questions per run: where did the tokens go, and where did the wall clock go? Then it adds a deterministic budget breaker you can test.

Why time-ordered traces mislead you

A trace UI sorts by started_at. That order is correct and useless for cost questions.

• A 180-second test run looks like the problem. It may cost zero tokens.
• A 4-second model call can hold half the run's tokens.
• A retried tool call doubles a line item that looks free.

You need two rankings, not one sorted list. Build both from the same spans.

The minimum span schema

Six fields carry the whole ledger. Anything else is optional.

Field	Type	Why the ledger needs it
span_id	string	Deduplicates on replay
parent_id	string | null	Separates sub-agent spend from the root
seq	integer	Gives a deterministic iteration order
kind	model \	tool \
started_at / ended_at	ISO-8601 UTC	Computes wall clock per span
prompt_tokens / completion_tokens	integer	Records cost; tools must log 0, not omit
retry_of	string | null	Marks a repeated attempt

If your tracer cannot emit per-span token counts, stop here. Aggregate usage per run is not enough to attribute anything.

Artifact 1: the ledger script

Save the fixture and the script in one directory. The fixture is six spans, one retry, one slow tool.

{"span_id":"s1","parent_id":null,"seq":1,"name":"planner","kind":"model","started_at":"2026-10-18T01:00:00Z","ended_at":"2026-10-18T01:00:06Z","prompt_tokens":900,"completion_tokens":300,"status":"ok","retry_of":null}
{"span_id":"s2","parent_id":"s1","seq":2,"name":"repo.search","kind":"tool","started_at":"2026-10-18T01:00:06Z","ended_at":"2026-10-18T01:00:07Z","prompt_tokens":0,"completion_tokens":0,"status":"ok","retry_of":null}
{"span_id":"s3","parent_id":"s2","seq":3,"name":"reader","kind":"model","started_at":"2026-10-18T01:00:07Z","ended_at":"2026-10-18T01:00:15Z","prompt_tokens":3200,"completion_tokens":800,"status":"ok"}
{"span_id":"s4","parent_id":"s2","seq":4,"name":"repo.search","kind":"tool","started_at":"2026-10-18T01:00:15Z","ended_at":"2026-10-18T01:00:16Z","prompt_tokens":0,"completion_tokens":0,"status":"ok","retry_of":"s2"}
{"span_id":"s5","parent_id":null,"seq":5,"name":"test.run","kind":"tool","started_at":"2026-10-18T01:00:16Z","ended_at":"2026-10-18T01:03:16Z","prompt_tokens":0,"completion_tokens":0,"status":"ok"}
{"span_id":"s6","parent_id":"s3","seq":6,"name":"summarize","kind":"model","started_at":"2026-10-18T01:03:16Z","ended_at":"2026-10-18T01:03:21Z","prompt_tokens":2600,"completion_tokens":400,"status":"ok"}

# ledger.py
import json
from collections import defaultdict
from datetime import datetime
from pathlib import Path


def ts(value: str) -> datetime:
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def read_spans(path) -> list:
    spans = []
    for lineno, raw in enumerate(Path(path).read_text().splitlines(), 1):
        raw = raw.strip()
        if not raw:
            continue
        try:
            spans.append(json.loads(raw))
        except json.JSONDecodeError as exc:
            raise ValueError(f"{path}:{lineno}: {exc}") from exc
    return sorted(spans, key=lambda s: s["seq"])


def span_row(span) -> dict:
    ms = int((ts(span["ended_at"]) - ts(span["started_at"])).total_seconds() * 1000)
    tokens = span.get("prompt_tokens", 0) + span.get("completion_tokens", 0)
    return {
        "name": span["name"],
        "kind": span["kind"],
        "ms": ms,
        "tokens": tokens,
        "failed": span.get("status", "ok") != "ok",
        "retry": bool(span.get("retry_of")),
    }


def ledger(path):
    rows = [span_row(s) for s in read_spans(path)]
    total_ms = sum(r["ms"] for r in rows) or 1
    total_tokens = sum(r["tokens"] for r in rows) or 1
    groups = defaultdict(
        lambda: {"calls": 0, "ms": 0, "tokens": 0, "retries": 0, "failed": 0}
    )
    for r in rows:
        g = groups[(r["kind"], r["name"])]
        g["calls"] += 1
        g["ms"] += r["ms"]
        g["tokens"] += r["tokens"]
        g["retries"] += int(r["retry"])
        g["failed"] += int(r["failed"])
    out = []
    for (kind, name), g in groups.items():
        out.append({**g, "kind": kind, "name": name,
                    "token_share": round(100 * g["tokens"] / total_tokens, 1),
                    "time_share": round(100 * g["ms"] / total_ms, 1)})
    return sorted(out, key=lambda r: r["tokens"], reverse=True), total_ms, total_tokens


if __name__ == "__main__":
    rows, total_ms, total_tokens = ledger("trace.jsonl")
    print(f"run total: {total_tokens} tokens, {total_ms} ms")
    for r in rows:
        print(
            f"{r['kind']:<5} {r['name']:<12} calls={r['calls']} "
            f"tokens={r['tokens']} ({r['token_share']}%) "
            f"ms={r['ms']} ({r['time_share']}%) retries={r['retries']}"
        )

Expected output for the fixture above (values follow from the six spans; run it to confirm):

run total: 8200 tokens, 201000 ms
model reader       calls=1 tokens=4000 (48.8%) ms=8000   (4.0%)  retries=0
tool  repo.search  calls=2 tokens=0    (0.0%)  ms=2000   (1.0%)  retries=1
model summarize    calls=1 tokens=3000 (36.6%) ms=5000   (2.5%)  retries=0
model planner      calls=1 tokens=1200 (14.6%) ms=6000   (3.0%)  retries=0
tool  test.run     calls=1 tokens=0    (0.0%)  ms=180000 (89.6%) retries=0

The two rankings point at different spans. reader owns 48.8% of tokens. test.run owns 89.6% of wall clock and zero tokens. Cutting the slow tool saves nothing on cost.

Artifact 2: a budget breaker you can test

The ledger is retrospective. Add one forward-looking rule before the next dispatch.

Rule: if committed + estimate > cap, do not dispatch the span. Write the decision and stop the run.

# breaker.py
import json
from pathlib import Path
from ledger import read_spans


def next_span_allowed(committed: int, cap: int, est: int):
    projected = committed + est
    if projected > cap:
        return False, f"projected {projected} > cap {cap}"
    return True, f"projected {projected} <= cap {cap}"


def walk(path, cap: int, out_path):
    committed, last_model = 0, 0
    lines = []
    for span in read_spans(path):
        tokens = span.get("prompt_tokens", 0) + span.get("completion_tokens", 0)
        est = span.get("est_tokens", last_model if span["kind"] == "model" else 0)
        ok, reason = next_span_allowed(committed, cap, est)
        lines.append({"seq": span["seq"], "name": span["name"],
                      "action": "dispatch" if ok else "abort",
                      "committed": committed, "est": est, "reason": reason})
        if not ok:
            break
        committed += tokens
        if span["kind"] == "model":
            last_model = tokens
    Path(out_path).write_text("\n".join(json.dumps(l) for l in lines) + "\n")
    return lines

The estimator is crude on purpose. It uses the last model span, or an explicit est_tokens field when the caller knows better. Crude and deterministic beats clever and untestable.

Artifact 3: the regression set

Five tests keep the ledger honest. They run in under a second and need no network.

# test_ledger.py
import json
import pytest
from ledger import ledger, read_spans
from breaker import walk


def test_totals():
    _, total_ms, total_tokens = ledger("trace.jsonl")
    assert (total_tokens, total_ms) == (8200, 201_000)


def test_rankings_differ():
    rows, _, _ = ledger("trace.jsonl")
    assert rows[0]["name"] == "reader"
    slowest = max(rows, key=lambda r: r["ms"])
    assert slowest["name"] == "test.run"
    assert slowest["tokens"] == 0


def test_retry_is_counted_not_hidden():
    rows, _, _ = ledger("trace.jsonl")
    search = next(r for r in rows if r["name"] == "repo.search")
    assert search["calls"] == 2 and search["retries"] == 1


def test_breaker_aborts_before_dispatch(tmp_path):
    out = tmp_path / "budget.jsonl"
    lines = walk("trace.jsonl", cap=6000, out_path=out)
    assert lines[-1]["action"] == "abort"
    assert lines[-1]["seq"] == 6
    assert all(l["action"] == "dispatch" for l in lines[:-1])
    assert json.loads(out.read_text().splitlines()[-1])["action"] == "abort"


def test_bad_line_names_the_line(tmp_path):
    bad = tmp_path / "bad.jsonl"
    bad.write_text('{"seq": 1}\nnot json\n')
    with pytest.raises(ValueError, match="bad.jsonl:2"):
        read_spans(bad)

At cap=6000 the walk aborts at seq 6 with projected 9200 > cap 6000. The run keeps 5200 committed tokens and skips the 3000-token summary.

The debug loop, five steps

1. Capture the trace with seq, kind, and per-span tokens.
2. Run the ledger. Read both rankings before forming a hypothesis.
3. Name the top token span and the top time span. If they differ, you have two bugs, not one.
4. Cut one thing: shorten the prompt, cache the retrieval, or drop the retry.
5. Re-run. Diff the two ledgers. The delta is the evidence.

Step 5 is where most teams stop early. A cheaper run with a different failure mode is not a win. Compare the failed column too.

Where free access fits, and where it does not

The loop above reruns a lot. Every cut means another replay with the same fixture. That is exactly the shape of work where free model access and a free server option reduce friction: the replay runs while you sleep, and the ledger diff is waiting.

MonkeyCode offers free model access and a free server option, per the project's own description. Check the current limits on its site before you size a batch around them; availability and caps are the operator's to state, not a blog post's.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One more caveat. Do not point the breaker at a paid key with a production budget. Run it in a sandbox first, and keep the decision log next to the trace.

If you want to try the loop without buying tokens first, the free tier is the low-risk place to start.

Limitations and who should skip this

• Aggregate-only providers. If your API returns one usage number per request tree, per-span attribution is fiction.
• Single-call scripts. One model call has no ranking to compare. Skip the ledger.
• Streaming partials. A span that never closes has no ended_at. Emit a sentinel on timeout or the table lies.
• Nested sub-agents. Use parent_id to build a tree. Flat sums double-count when a parent includes child tokens.
• Sub-second tools. Millisecond rounding will dominate. Track counts, not shares.

The ledger is not a dashboard. It is a five-line answer to "which span do I cut?" That is usually enough.

What to keep

Two rankings, one trace. Token share and time share disagree more often than intuition suggests.

Record seq, kind, and tokens per span. Everything else is arithmetic you can write in forty lines.

Test the breaker on a fixture, not on a live budget. A rule you cannot reproduce is a rule you cannot trust.