Stalled Agents Burn Tokens Quietly: A Progress Ledger for Traces

Most agent failures don't announce themselves. The run ends green, the trace is huge, and the workspace is exactly where it started. The number worth gating on is not tokens per run. It is tokens per novel change.

Why span timing misleads you

A span tells you that a tool ran and how long it took. It does not tell you whether the run moved. An agent can call read_file, grep, and apply_patch eleven times in a row, produce a spotless trace, and never change a line of code — and every one of those spans looks healthy on a latency chart.

That gap matters more now because cheap model output is flooding into agent loops. The current DEV discussion is mostly about who sits between you and the model; the unglamorous counterpart is accounting for what each loop iteration actually bought. If a loop costs almost nothing per call, nobody notices that it is looping. Cost per call hides waste. Cost per outcome exposes it.

Here is the rule I keep coming back to. A span is progress only if it changes the state of the world in a way no earlier span already changed.

The progress ledger

You already have the raw material: a JSONL trace where each span records its kind, its token count, and the unified diff it produced (empty for read-only spans). Reduce each diff to a coarse key — file path, added lines, removed lines — and keep a set of keys seen so far.

A span is novel when it contributes at least one key the set has never held. Everything else is churn, and churn is where the money goes. The script below is a proposal; it uses only the Python standard library and has not been run against production traces, so treat the thresholds as starting points you will re-tune.

# progress_ledger.py — proposed reducer; standard library only
import json, sys

def churn_keys(diff_text):
    """Map a unified diff to (path, added, removed) keys."""
    keys, path, plus, minus = set(), None, 0, 0

    def flush():
        if path and (plus or minus):
            keys.add((path, plus, minus))

    for line in diff_text.splitlines():
        if line.startswith('+++ '):
            flush()
            path, plus, minus = line[4:].strip(), 0, 0
        elif line.startswith('+') and not line.startswith('+++'):
            plus += 1
        elif line.startswith('-') and not line.startswith('---'):
            minus += 1
    flush()
    return keys

def reduce_run(spans):
    seen, novel, stall, stall_spans = set(), 0, 0, 0
    for s in spans:
        fresh = churn_keys(s.get('diff') or '') - seen
        if fresh:
            seen |= fresh
            novel += 1
        elif s['kind'] == 'tool':
            stall += s['tokens']
            stall_spans += 1
    total = sum(s['tokens'] for s in spans)
    return {
        'spans': len(spans),
        'novel_spans': novel,
        'stall_tokens': stall,
        'stall_share': round(stall / total, 3) if total else 0.0,
        'tokens_per_novel_span': round(total / novel, 1) if novel else None,
    }

if __name__ == '__main__':
    spans = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
    print(json.dumps(reduce_run(spans), indent=2))

Run it with python progress_ledger.py traces/run-2026-10-27.jsonl and read three fields: novel_spans, stall_share, tokens_per_novel_span. A run with one novel span and a 0.6 stall share is not a slow run. It is a run that spent most of its budget re-reading its own mistakes.

One caveat before you trust the number: reformatting churn counts as novelty here, because the key is line counts rather than content. Normalise whitespace in the diff before hashing if your agents reformat aggressively.

Pin the behaviour with a test

A reducer you never test is a rumour. Five synthetic spans are enough to lock the semantics, including the important case where a patch is applied twice.

# test_ledger.py — run with `python -m pytest -q`
from progress_ledger import reduce_run

SPANS = [
    {'idx': 0, 'kind': 'tool', 'tokens': 900,  'diff': ''},
    {'idx': 1, 'kind': 'tool', 'tokens': 850,  'diff': '+++ b/app.py\n+x = 1\n'},
    {'idx': 2, 'kind': 'tool', 'tokens': 800,  'diff': '+++ b/app.py\n+x = 1\n'},
    {'idx': 3, 'kind': 'tool', 'tokens': 780,  'diff': ''},
    {'idx': 4, 'kind': 'tool', 'tokens': 4000, 'diff': '+++ b/app.py\n+x = 1\n+y = 2\n'},
]

def test_repeat_patch_is_not_progress():
    r = reduce_run(SPANS)
    assert r['novel_spans'] == 2, r
    assert r['stall_tokens'] == 1580, r
    assert r['stall_share'] == 0.216, r
    assert r['tokens_per_novel_span'] == 3665.0, r

def test_empty_run_does_not_divide_by_zero():
    assert reduce_run([])['tokens_per_novel_span'] is None

The second assertion is the whole point of the exercise. Span 1 writes x = 1; span 2 writes the identical patch; span 3 writes nothing. Two spans, 1,580 tokens, zero new information. Without the ledger, span 2 looks like work.

Gate the loop in CI

Once the numbers are stable, the gate is a one-liner. Compare a candidate run against a stored baseline and fail when the stall share grows past your tolerance.

python progress_ledger.py traces/candidate.jsonl > /tmp/c.json
python - <<'PY'
import json
c = json.load(open('/tmp/c.json'))
assert c['stall_share'] < 0.3, f"stall share regression: {c}"
assert c['tokens_per_novel_span'] < 5000, f"cost per change regression: {c}"
print('ok', c)
PY

Keep the baseline file next to the trace. A regression test that cannot show you the previous number is just a mood.

Where free capacity fits, and where it does not

Cheap or free inference is exactly what makes the collector worth running, because the collector is what stops "free" from meaning "unmeasured". If you want a place to run it, the MonkeyCode open-source project is one option: its operator describes free model access and a free server option, with 10 million free tokens stated at the time of writing. I have not verified quota permanence or capacity, so confirm the current terms on the project page before you design around them. The reducer above needs nothing beyond a Python interpreter and a trace directory, so the free server is a reasonable host for the collector while your agent runs elsewhere.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A short decision table, since the honest answer is not always "use the free tier":

Situation	Sensible choice
Iterating on trace tooling, reducer tests, dashboards	Free model access and free server are enough
Runs whose failure costs more than the tokens saved	Paid, pinned model version, reproducible
Prompts or traces containing customer data	Self-hosted or on-prem only, regardless of price
Needing guaranteed throughput or uptime	Do not build on any free tier, including this one

Limitations and who should skip this

This method only sees what your trace records. If tool spans do not carry diffs, or if progress happens outside the workspace (a database migration, an API call that mutates remote state), the ledger will call real work churn. Instrument those side effects as diff-like keys or exclude those runs from the gate.

Skip it entirely if your runs are single-shot with no tool loop — there is nothing to stall. Skip it too if you cannot yet produce stable traces; a progress ledger on top of noisy spans buys you confident wrong numbers.

The cheapest loop is the one that changes something every time it runs. Measure that, and the rest of the cost conversation gets a lot shorter. If you try the collector against free-tier traffic, the MonkeyCode project page is where to start.