DEV Community: Sam Sun

Stalled Agents Burn Tokens Quietly: A Progress Ledger for Traces

Sam Sun — Tue, 27 Oct 2026 13:39:37 +0000

Most agent failures don't announce themselves. The run ends green, the trace is huge, and the workspace is exactly where it started. The number worth gating on is not tokens per run. It is tokens per novel change.

Why span timing misleads you

A span tells you that a tool ran and how long it took. It does not tell you whether the run moved. An agent can call read_file, grep, and apply_patch eleven times in a row, produce a spotless trace, and never change a line of code — and every one of those spans looks healthy on a latency chart.

That gap matters more now because cheap model output is flooding into agent loops. The current DEV discussion is mostly about who sits between you and the model; the unglamorous counterpart is accounting for what each loop iteration actually bought. If a loop costs almost nothing per call, nobody notices that it is looping. Cost per call hides waste. Cost per outcome exposes it.

Here is the rule I keep coming back to. A span is progress only if it changes the state of the world in a way no earlier span already changed.

The progress ledger

You already have the raw material: a JSONL trace where each span records its kind, its token count, and the unified diff it produced (empty for read-only spans). Reduce each diff to a coarse key — file path, added lines, removed lines — and keep a set of keys seen so far.

A span is novel when it contributes at least one key the set has never held. Everything else is churn, and churn is where the money goes. The script below is a proposal; it uses only the Python standard library and has not been run against production traces, so treat the thresholds as starting points you will re-tune.

# progress_ledger.py — proposed reducer; standard library only
import json, sys

def churn_keys(diff_text):
    """Map a unified diff to (path, added, removed) keys."""
    keys, path, plus, minus = set(), None, 0, 0

    def flush():
        if path and (plus or minus):
            keys.add((path, plus, minus))

    for line in diff_text.splitlines():
        if line.startswith('+++ '):
            flush()
            path, plus, minus = line[4:].strip(), 0, 0
        elif line.startswith('+') and not line.startswith('+++'):
            plus += 1
        elif line.startswith('-') and not line.startswith('---'):
            minus += 1
    flush()
    return keys

def reduce_run(spans):
    seen, novel, stall, stall_spans = set(), 0, 0, 0
    for s in spans:
        fresh = churn_keys(s.get('diff') or '') - seen
        if fresh:
            seen |= fresh
            novel += 1
        elif s['kind'] == 'tool':
            stall += s['tokens']
            stall_spans += 1
    total = sum(s['tokens'] for s in spans)
    return {
        'spans': len(spans),
        'novel_spans': novel,
        'stall_tokens': stall,
        'stall_share': round(stall / total, 3) if total else 0.0,
        'tokens_per_novel_span': round(total / novel, 1) if novel else None,
    }

if __name__ == '__main__':
    spans = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
    print(json.dumps(reduce_run(spans), indent=2))

Run it with python progress_ledger.py traces/run-2026-10-27.jsonl and read three fields: novel_spans, stall_share, tokens_per_novel_span. A run with one novel span and a 0.6 stall share is not a slow run. It is a run that spent most of its budget re-reading its own mistakes.

One caveat before you trust the number: reformatting churn counts as novelty here, because the key is line counts rather than content. Normalise whitespace in the diff before hashing if your agents reformat aggressively.

Pin the behaviour with a test

A reducer you never test is a rumour. Five synthetic spans are enough to lock the semantics, including the important case where a patch is applied twice.

# test_ledger.py — run with `python -m pytest -q`
from progress_ledger import reduce_run

SPANS = [
    {'idx': 0, 'kind': 'tool', 'tokens': 900,  'diff': ''},
    {'idx': 1, 'kind': 'tool', 'tokens': 850,  'diff': '+++ b/app.py\n+x = 1\n'},
    {'idx': 2, 'kind': 'tool', 'tokens': 800,  'diff': '+++ b/app.py\n+x = 1\n'},
    {'idx': 3, 'kind': 'tool', 'tokens': 780,  'diff': ''},
    {'idx': 4, 'kind': 'tool', 'tokens': 4000, 'diff': '+++ b/app.py\n+x = 1\n+y = 2\n'},
]

def test_repeat_patch_is_not_progress():
    r = reduce_run(SPANS)
    assert r['novel_spans'] == 2, r
    assert r['stall_tokens'] == 1580, r
    assert r['stall_share'] == 0.216, r
    assert r['tokens_per_novel_span'] == 3665.0, r

def test_empty_run_does_not_divide_by_zero():
    assert reduce_run([])['tokens_per_novel_span'] is None

The second assertion is the whole point of the exercise. Span 1 writes x = 1; span 2 writes the identical patch; span 3 writes nothing. Two spans, 1,580 tokens, zero new information. Without the ledger, span 2 looks like work.

Gate the loop in CI

Once the numbers are stable, the gate is a one-liner. Compare a candidate run against a stored baseline and fail when the stall share grows past your tolerance.

python progress_ledger.py traces/candidate.jsonl > /tmp/c.json
python - <<'PY'
import json
c = json.load(open('/tmp/c.json'))
assert c['stall_share'] < 0.3, f"stall share regression: {c}"
assert c['tokens_per_novel_span'] < 5000, f"cost per change regression: {c}"
print('ok', c)
PY

Keep the baseline file next to the trace. A regression test that cannot show you the previous number is just a mood.

Where free capacity fits, and where it does not

Cheap or free inference is exactly what makes the collector worth running, because the collector is what stops "free" from meaning "unmeasured". If you want a place to run it, the MonkeyCode open-source project is one option: its operator describes free model access and a free server option, with 10 million free tokens stated at the time of writing. I have not verified quota permanence or capacity, so confirm the current terms on the project page before you design around them. The reducer above needs nothing beyond a Python interpreter and a trace directory, so the free server is a reasonable host for the collector while your agent runs elsewhere.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A short decision table, since the honest answer is not always "use the free tier":

Situation	Sensible choice
Iterating on trace tooling, reducer tests, dashboards	Free model access and free server are enough
Runs whose failure costs more than the tokens saved	Paid, pinned model version, reproducible
Prompts or traces containing customer data	Self-hosted or on-prem only, regardless of price
Needing guaranteed throughput or uptime	Do not build on any free tier, including this one

Limitations and who should skip this

This method only sees what your trace records. If tool spans do not carry diffs, or if progress happens outside the workspace (a database migration, an API call that mutates remote state), the ledger will call real work churn. Instrument those side effects as diff-like keys or exclude those runs from the gate.

Skip it entirely if your runs are single-shot with no tool loop — there is nothing to stall. Skip it too if you cannot yet produce stable traces; a progress ledger on top of noisy spans buys you confident wrong numbers.

The cheapest loop is the one that changes something every time it runs. Measure that, and the rest of the cost conversation gets a lot shorter. If you try the collector against free-tier traffic, the MonkeyCode project page is where to start.

Reconcile Logs, Tool Calls, and Diffs Before You Trust a Trace

Sam Sun — Mon, 26 Oct 2026 12:24:29 +0000

An agent trace is not one record. It is three records that never fully agree: the log line, the tool call, and the diff. The log says what the program believed was happening. The tool record says what the runtime actually dispatched. The diff says what the filesystem finally accepted.

When those three disagree, you get a bug class that no single source can show you. A green log with a red diff is an unlogged write. A dispatched call with an empty diff is a lie about work done. A log that announces a retry the runtime never issued is an incident note that will mislead the next reader. Reconcile the three before you read anything else in the trace.

The fixture first, because a harness you cannot reproduce is a rumor

Start with a trace you can hand to a colleague. This one has a clean read, a clean write, and two planted disagreements.

{
  "run_id": "r-42",
  "logs": [
    {"t": 0.00, "msg": "plan ready"},
    {"t": 0.40, "msg": "tool read_file start"},
    {"t": 0.90, "msg": "tool read_file ok"},
    {"t": 1.20, "msg": "retrying write_file"}
  ],
  "tool_calls": [
    {"name": "read_file", "start": 0.40, "end": 0.90, "args": {"path": "src/app.py"}},
    {"name": "write_file", "start": 1.10, "end": 1.60, "args": {"path": "src/app.py"}}
  ],
  "diffs": [
    {"path": "src/app.py", "added": 4, "removed": 0},
    {"path": "tests/test_app.py", "added": 2, "removed": 1}
  ]
}

Read it as a reviewer and you already feel the friction. Something edited a test file that no tool call claims to have touched. Something said "retrying" while only one write exists. The point of the harness is to make that friction mechanical, so a nightly job catches it instead of a tired human at 11pm.

A harness small enough to audit

The join is plain set arithmetic plus one regex, and it returns a list rather than printing as it goes, which keeps it testable.

#!/usr/bin/env python3
"""reconcile.py - join logs, tool calls, and diffs into one finding list."""
import json
import re
import sys

MUTATING = {"write_file", "edit_file", "delete_file", "apply_patch"}
BLOCKING = {"silent_write", "phantom_call"}


def reconcile(trace):
    findings = []
    calls = [c for c in trace["tool_calls"] if c["name"] in MUTATING]
    written = {c["args"].get("path"): c for c in calls}
    diff_paths = {d["path"]: d for d in trace["diffs"]}

    for path, d in diff_paths.items():
        if path not in written:
            findings.append({"kind": "silent_write", "path": path,
                             "detail": f'+{d["added"]}/-{d["removed"]}, no mutating call'})

    for path, c in written.items():
        if path not in diff_paths:
            findings.append({"kind": "phantom_call", "path": path,
                             "detail": f'{c["name"]} dispatched, nothing changed'})

    for line in trace["logs"]:
        m = re.search(r"retry\w*\s+(\w+)", line["msg"])
        if not m:
            continue
        name = m.group(1)
        seen = sum(1 for c in trace["tool_calls"] if c["name"] == name)
        if seen < 2:
            findings.append({"kind": "retry_unrecorded", "path": name,
                             "detail": f'log@{line["t"]:.2f} claims retry of {name}; {seen} call recorded'})
    return findings


def main(path):
    findings = reconcile(json.load(open(path)))
    for f in findings:
        print(f'{f["kind"]:<16} {f["path"]:<20} {f["detail"]}')
    blocking = [f for f in findings if f["kind"] in BLOCKING]
    print(f'{len(findings)} findings ({len(blocking)} blocking)')
    return 1 if blocking else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it against the fixture and the disagreements surface in a fixed order, which matters if you plan to diff the output between runs.

$ python reconcile.py trace.json
silent_write     tests/test_app.py    +2/-1, no mutating call
retry_unrecorded write_file           log@1.20 claims retry of write_file; 1 call recorded
2 findings (1 blocking)
$ echo $?
1

The exit code is the contract. Advisory findings (an unexplained retry) return 0 and inform. Blocking findings (a mutation nobody dispatched) return 1 and stop the pipeline. That split is the whole design: you want the retry note in the report, and you want the silent write to fail the run.

Four disagreements worth naming

Kind	Shape of the evidence	Blocking
`silent_write`	Diff path with no mutating call	Yes
`phantom_call`	Mutating call with no diff at its declared path	Yes
`retry_unrecorded`	Log claims a retry the call list cannot confirm	No
`redirect_suspect`	Exactly one unexplained write and one unexplained call, paired by order	Review

The last row is a heuristic and should stay labelled as one. Pairing a lone orphan call with a lone orphan diff is a reasonable guess, not proof. Keep it advisory until you have seen it be right on your own traces.

What this cannot see

Read-only tools are exempt by the MUTATING set, so a shell tool that mutates files through sed -i will produce a false silent_write. Path normalization bites too: symlinks, absolute versus relative paths, and case differences all break the join. Concurrent runs sharing a working tree will interleave diffs and produce findings that are artifacts of scheduling rather than agent behavior.

In-memory state, network side effects, and anything that never touches the working tree are simply outside the model. The harness answers one narrow question — do these three records describe the same run — and it answers it only as well as your tool registry is honest.

The fixture test comes next, because a harness with no test is just another opinion. Ten lines of pytest pinning the exact finding set is enough.

from reconcile import reconcile
import json


def test_fixture_findings():
    trace = json.load(open("trace.json"))
    kinds = {(f["kind"], f["path"]) for f in reconcile(trace)}
    assert kinds == {("silent_write", "tests/test_app.py"),
                     ("retry_unrecorded", "write_file")}

Running it where the traces already land

A reconciliation step earns its keep as a scheduled job, not a manual script. Once a night, over yesterday's traces, writing findings to disk and letting the exit code drive a notification is the smallest useful version. I run that nightly job on the free server option of MonkeyCode rather than on a laptop, so the harness sits next to the trace store instead of depending on someone remembering to run it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free-tier terms and limits change, so check the project's own page for the current offer rather than trusting any number quoted in a blog post.

Free model access is a good fit for exactly one narrow job here: mapping tool names from different SDKs (str_replace_editor, apply_patch, write) onto your MUTATING taxonomy. That mapping is boring, it breaks whenever you add an integration, and it must never be trusted blindly. Sketch, not production code:

# pseudocode: model proposes a mapping, the frozen fixture verifies it
proposed = classify_tool_names(unmapped_names)          # model call
assert proposed == FROZEN_EXPECTED                      # test gate
TAXONOMY.update(proposed if proposed else RULE_MAP)     # rule map fallback

The rule map is the default. The model fills gaps between releases, and the frozen fixture decides whether the answer ships. If the assertion fails, you fall back and read the diff — which is the same discipline the harness itself is built on.

Who should skip this

Single-turn scripts with no file mutations will produce empty finding lists forever. Teams already capturing full OpenTelemetry spans plus a diff-per-tool-call record have solved this at the source, and a reconciliation pass on top is redundant ceremony. And if your traces are large enough that a nightly Python pass over them is the expensive part, fix retention first — reconciliation over a sampled week usually tells you everything a full month would.

If you keep a trace store and have no reconciliation step, run this against your last twenty runs before you change any prompt. The disagreements you find tend to be older than your current bug.

Reconcile Logs, Tool Calls, and Diffs Before You Trust a Trace

Sam Sun — Mon, 26 Oct 2026 12:16:52 +0000

The fixture first, because a harness you cannot reproduce is a rumor

Start with a trace you can hand to a colleague. This one has a clean read, a clean write, and two planted disagreements.

{
  "run_id": "r-42",
  "logs": [
    {"t": 0.00, "msg": "plan ready"},
    {"t": 0.40, "msg": "tool read_file start"},
    {"t": 0.90, "msg": "tool read_file ok"},
    {"t": 1.20, "msg": "retrying write_file"}
  ],
  "tool_calls": [
    {"name": "read_file", "start": 0.40, "end": 0.90, "args": {"path": "src/app.py"}},
    {"name": "write_file", "start": 1.10, "end": 1.60, "args": {"path": "src/app.py"}}
  ],
  "diffs": [
    {"path": "src/app.py", "added": 4, "removed": 0},
    {"path": "tests/test_app.py", "added": 2, "removed": 1}
  ]
}

A harness small enough to audit

The join is plain set arithmetic plus one regex, and it returns a list rather than printing as it goes, which keeps it testable.

#!/usr/bin/env python3
"""reconcile.py - join logs, tool calls, and diffs into one finding list."""
import json
import re
import sys

MUTATING = {"write_file", "edit_file", "delete_file", "apply_patch"}
BLOCKING = {"silent_write", "phantom_call"}


def reconcile(trace):
    findings = []
    calls = [c for c in trace["tool_calls"] if c["name"] in MUTATING]
    written = {c["args"].get("path"): c for c in calls}
    diff_paths = {d["path"]: d for d in trace["diffs"]}

    for path, d in diff_paths.items():
        if path not in written:
            findings.append({"kind": "silent_write", "path": path,
                             "detail": f'+{d["added"]}/-{d["removed"]}, no mutating call'})

    for path, c in written.items():
        if path not in diff_paths:
            findings.append({"kind": "phantom_call", "path": path,
                             "detail": f'{c["name"]} dispatched, nothing changed'})

    for line in trace["logs"]:
        m = re.search(r"retry\w*\s+(\w+)", line["msg"])
        if not m:
            continue
        name = m.group(1)
        seen = sum(1 for c in trace["tool_calls"] if c["name"] == name)
        if seen < 2:
            findings.append({"kind": "retry_unrecorded", "path": name,
                             "detail": f'log@{line["t"]:.2f} claims retry of {name}; {seen} call recorded'})
    return findings


def main(path):
    findings = reconcile(json.load(open(path)))
    for f in findings:
        print(f'{f["kind"]:<16} {f["path"]:<20} {f["detail"]}')
    blocking = [f for f in findings if f["kind"] in BLOCKING]
    print(f'{len(findings)} findings ({len(blocking)} blocking)')
    return 1 if blocking else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it against the fixture and the disagreements surface in a fixed order, which matters if you plan to diff the output between runs.

$ python reconcile.py trace.json
silent_write     tests/test_app.py    +2/-1, no mutating call
retry_unrecorded write_file           log@1.20 claims retry of write_file; 1 call recorded
2 findings (1 blocking)
$ echo $?
1

Four disagreements worth naming

Kind	Shape of the evidence	Blocking
`silent_write`	Diff path with no mutating call	Yes
`phantom_call`	Mutating call with no diff at its declared path	Yes
`retry_unrecorded`	Log claims a retry the call list cannot confirm	No
`redirect_suspect`	Exactly one unexplained write and one unexplained call, paired by order	Review

What this cannot see

The fixture test comes next, because a harness with no test is just another opinion. Ten lines of pytest pinning the exact finding set is enough.

from reconcile import reconcile
import json


def test_fixture_findings():
    trace = json.load(open("trace.json"))
    kinds = {(f["kind"], f["path"]) for f in reconcile(trace)}
    assert kinds == {("silent_write", "tests/test_app.py"),
                     ("retry_unrecorded", "write_file")}

Running it where the traces already land

# pseudocode: model proposes a mapping, the frozen fixture verifies it
proposed = classify_tool_names(unmapped_names)          # model call
assert proposed == FROZEN_EXPECTED                      # test gate
TAXONOMY.update(proposed if proposed else RULE_MAP)     # rule map fallback

Who should skip this

If you keep a trace store and have no reconciliation step, run this against your last twenty runs before you change any prompt. The disagreements you find tend to be older than your current bug.

Label Spans With a Free Model, Then Gate the Labels on Agreement

Sam Sun — Tue, 20 Oct 2026 10:19:40 +0000

Most trace tooling answers what happened. Almost none answers which 40 spans of this 3,000-span run deserve a human's next hour. A cheap classifier can answer that — but only if you measure it before you trust it. The loop below builds bounded span cards, labels them with a free model endpoint, gates those labels against a hand-labeled slice, and only then lets them reorder your debug queue.

This week's AI feed is mostly capability talk. The unglamorous counterpart is verification: a labeler you have not tested is a random number generator with good manners.

Start from your own failure taxonomy

Do not borrow a label set from an article, including this one. Pull it from your last five postmortems. In my traces the recurring buckets are tool contract breaks (the tool returned a shape the prompt did not expect), prompt drift (the model stopped following a constraint mid-run), timeouts that were retried silently, and context loss after a handoff. Four buckets and an unknown escape hatch is enough to triage with.

unknown is not a failure mode. It is the honest answer when the card carries no evidence, and its rate is the first number you should look at. A labeler that answers unknown 60% of the time is telling you your cards are too thin, not that your agent is mysterious.

Card first, prompt second

A span record is usually 2–20 KB of JSON. Sending that verbatim to any model is wasteful and it invites the model to summarize rather than classify. The card is a fixed-byte projection: identity, kind, status, duration, and a truncated slice of input-to-output text.

import json

CARD_TEXT_BYTES = 320

def card(span):
    text = (span.get("input_summary") or "") + " -> " + (span.get("output_summary") or "")
    text = text.encode("utf-8")[:CARD_TEXT_BYTES].decode("utf-8", "ignore")
    return {
        "id": span["span_id"],
        "kind": span["kind"],        # "llm" | "tool" | "handoff"
        "tool": span.get("tool_name"),
        "status": span["status"],    # "ok" | "error" | "timeout"
        "dur_ms": span["dur_ms"],
        "text": text,
    }

Truncation is deliberate and it must be visible. If a span is misclassified because the evidence was cut off at byte 320, raising the byte budget should fix it — that is a falsifiable hypothesis, which is more than most prompt edits give you.

One span in, one JSON object out

The client is a callable, not an SDK import. That keeps the harness portable across whatever endpoint you point it at and keeps the retry policy in your code where you can test it.

LABELS = ["tool_contract", "prompt_drift", "timeout", "context_loss", "unknown"]

PROMPT = (
    "You label one agent span. Reply with JSON only: "
    '{"label": <one of ' + ", ".join(LABELS) + '>, "why": "<=12 words"}. '
    "Use evidence in the card only. If the card lacks evidence, answer unknown."
)

def label(llm, span, retries=2):
    c = card(span)
    msgs = [{"role": "user", "content": PROMPT + "\n\nCARD:\n" + json.dumps(c)}]
    for _ in range(retries + 1):
        try:
            raw = llm(msgs)                      # your client, one function
            obj = json.loads(raw[raw.index("{"): raw.rindex("}") + 1])
            if obj.get("label") in LABELS:
                return {"span_id": c["id"], "label": obj["label"], "why": obj.get("why", "")}
        except Exception:
            continue
    return {"span_id": c["id"], "label": "unknown", "why": "parse_fail"}

Parse failures collapse into unknown on purpose. A silent fallback that pretends to classify is worse than a bucket you can count. Run it over a sample rather than the whole trace: python -m triage.run --trace run.jsonl --sample 200 --out labels.jsonl.

The gate is the entire point

Hand-label 40–60 spans you have already debugged, once. Then measure the labeler against them, per class, not just overall.

def gate(gold, pred, floor=0.70):
    by = {}
    for sid, g in gold.items():
        p = pred.get(sid, "unknown")
        slot = by.setdefault(g, [0, 0])
        slot[0] += 1
        slot[1] += int(p == g)
    recall = {k: round(v[1] / v[0], 2) for k, v in by.items()}
    overall = round(sum(v[1] for v in by.values()) / len(gold), 2)
    return {"overall": overall, "recall": recall, "pass": overall >= floor}

The floor is a knob you calibrate, not a law. What matters is the failure direction: a labeler that misses timeout sends you chasing prompts, while one that over-fires tool_contract floods the queue with false leads. Look at the per-class recall before you look at the average, and freeze the gold set so future prompt edits are comparable. The snippets here are illustrative — I have not published numbers for them, and you should not adopt anyone's thresholds without running your own gold set.

Where the free tier actually fits

Labeling is high-volume, low-stakes, and its output is a hint rather than a verdict. That is the shape of work a free model endpoint absorbs well, and it is the reason I put labeling there instead of on the expensive path. MonkeyCode exposes free model access and a free server option; the operator describes a free token allowance in the ten-million range at the time of writing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Quotas, model line-ups, and server terms move, so verify the current page before you wire any of it into CI, and never treat a free allowance as a capacity guarantee. Redact before you send: strip bearer tokens, customer identifiers, and file contents from input_summary and output_summary at write time, not at card time. A span that never stored the secret cannot leak it to a labeler.

Step	Where it runs	Reason
Card building	Local	Deterministic, no egress, cheap to re-run
Labeling pass	Free model endpoint	High volume, low stakes, output is advisory
Agreement gate	Local	Needs your gold set; must stay reproducible
Full replay of shortlisted spans	Free server option	Long-running, needs sandbox egress you do not want on a laptop

The last row is where a free server earns its place: replay of a shortlisted span often needs to re-reach the same tools, and that is a job for a machine you can throw away. Keep the replayer's credentials scoped to a sandbox and log the replay's own spans separately, or you will merge two runs into one trace and lose the comparison.

Limitations, and who should skip this

If you debug fewer than a few hundred spans a week, read them. The harness costs more than it saves, and hand reading builds the intuition the taxonomy needs anyway. If your spans carry regulated data that cannot leave your network, the labeling pass has to stay local. If your tool schemas change weekly, the labeler drifts with them and the gate will fail often enough to be noise.

Agreement is not correctness. A labeler that matches your gold set has matched your judgment on that slice, and nothing more. Treat it as a queue sorter: it decides what you read first, never what you conclude.

If you want to try the loop without standing up infrastructure, the free model endpoint and free server option are the cheapest place to start — point the llm callable at the former, run the gate locally, and only move replay to the latter once the shortlist is stable.

Your Agent Trace Is Too Big: Budget Bytes per Span

Sam Sun — Mon, 19 Oct 2026 09:02:17 +0000

The scarce resource in agent observability is not spans. It is bytes.

A single read_file against a lockfile, one HTTP call to a chatty API, or one git diff over a generated directory can outweigh the other forty spans of a run combined. Picture a run whose span metadata totals 30 KB while one tool result carries 1.2 MB of JSON — those figures are illustrative, the shape is not. Store that verbatim and three things break at once: the trace viewer crawls, the disk on whatever box runs the agent fills up, and the signal you actually needed (which arguments went in, what shape came out) drowns in payload.

So budget bytes per span, and make the budget a test. This is the loop I keep coming back to when a trace gets fat.

Decide what earns verbatim storage

Two questions per payload, answered before the record is written rather than after: will you need exact bytes to reproduce the bug, and how large is it? The second question is cheap. The first one requires you to know what kind of tool you are instrumenting.

Payload kind	Keep inline	Spill to blob store	Why
Small tool arg / result (< 2 KB)	Full	No	Cheap, and exact bytes matter most here
File read	Head + tail + sha256	Yes, if over budget	You rarely debug the 900th line of a lockfile
HTTP response body	Head + tail + digest + status	Yes, if over budget	Status and headers explain more than the body
Model prompt / completion	Full, redacted upstream	No	Mid-payload truncation destroys reasoning context

Note the digest column. Every record carries a sha256 of the full payload even when the payload itself is truncated. That buys two things: deduplication, since identical file reads resolve to one blob, and a cheap equality check when you suspect two runs saw different bytes.

Capture inside a head/tail budget

import base64, hashlib, json, os, pathlib

STORE = pathlib.Path(os.environ.get("TRACE_BLOB_DIR", "./trace-blobs"))


def capture(raw: bytes, budget: int = 2048) -> dict:
    """Shrink one tool payload to a fixed-size record. Never lossless."""
    half = max(budget // 2, 0)
    digest = hashlib.sha256(raw).hexdigest()
    rec = {
        "sha256": digest,
        "bytes": len(raw),
        "truncated": len(raw) > budget,
        "head_b64": base64.b64encode(raw[:half]).decode(),
        "tail_b64": base64.b64encode(raw[-half:] if half else b"").decode(),
    }
    if rec["truncated"]:
        put_blob(digest, raw)
    return rec


def put_blob(digest: str, raw: bytes) -> None:
    STORE.mkdir(parents=True, exist_ok=True)
    path = STORE / digest
    if path.exists():
        return
    tmp = path.with_suffix(".tmp")
    tmp.write_bytes(raw)
    tmp.replace(path)


def rehydrate(rec: dict) -> bytes:
    path = STORE / rec["sha256"]
    if path.exists():
        return path.read_bytes()
    head = base64.b64decode(rec["head_b64"])
    tail = base64.b64decode(rec["tail_b64"])
    gap = rec["bytes"] - len(head) - len(tail)
    return head + f"<{gap} bytes omitted>".encode() + tail

Head answers "what is this object". Tail answers "how did it end", which is where stack traces, exit codes, and truncation markers usually live. The middle is what you sacrifice, and rehydrate states the size of that sacrifice instead of hiding it.

Two properties worth testing immediately. The blob store is content-addressed, so re-reading the same file ten times costs one file on disk. And the roundtrip must be exact when the payload fits:

def test_rehydrate_roundtrip():
    raw = b"x" * 10_000 + b"tail"
    rec = capture(raw, budget=512)
    assert rec["truncated"] is True
    assert rec["sha256"] == hashlib.sha256(raw).hexdigest()
    assert rehydrate(rec) == raw

When the blob is present, rehydrate returns the original bytes. When it has been garbage-collected, it returns a lossy but honest reconstruction. Both are debuggable; a 1.2 MB span in a web UI is not.

Make the budget a test, not a habit

A budget written in a document decays. A budget that fails a build survives. The test below measures the serialized size of each span record and names the largest offender.

import json, pathlib

LIMIT = 20_000  # bytes per span record; illustrative, tune per project


def span_bytes(span: dict) -> int:
    return len(json.dumps(span, separators=(",", ":"), sort_keys=True).encode())


def test_trace_byte_budget(trace: dict):
    sizes = sorted(
        ((span_bytes(s), s["name"]) for s in trace["spans"]), reverse=True
    )
    total = sum(n for n, _ in sizes)
    assert sizes[0][0] <= LIMIT, f"largest span {sizes[0][1]} is {sizes[0][0]} bytes"
    assert total <= LIMIT * 8, f"run total {total} bytes across {len(sizes)} spans"
    print(sizes[:3])

Run it the same way you run any other regression check:

python -m pytest tests/test_trace_bytes.py -q
du -sh trace-blobs | sort -h | tail -3

The first command tells you which tool is writing fat spans. The second tells you whether the blob store is growing faster than your retention policy can delete. When the assert fires, you get a name and a number, which is a decision: widen the budget for that tool, or fix the tool to return less.

Where this helps, and where it hurts

Truncation is lossy by design, and the loss lands in the middle of the payload — sometimes exactly where the malformed record sits. Digests prove that two runs differed; they never explain why. And a blob store without garbage collection is just a slower disk leak, so retention has to key off span status: keep blobs referenced by failed spans, drop the rest on a schedule you can defend.

Do not apply this to traces that are themselves a compliance or audit record. Those need exact bytes, immutable storage, and encryption at rest; a truncated record with a digest is not evidence. Likewise, if your tracing backend already offloads large payloads and gives you a stable handle back, you are duplicating work — use its offload instead. And redaction stays upstream of everything here: a blob is still storage, so secrets must be stripped before capture is called, not after.

One practical environment where the budget stops being theoretical: running the agent on a free server option, where the disk is yours to manage and nobody else will notice it filling. That is the case where per-span byte accounting pays for itself in the first week, and where a cheap summarization pass inside the debug loop is worth wiring up rather than skipping. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are what make that loop reachable without a budget conversation first; the byte ledger above is the part you keep either way.

Start by measuring one real trace you already have. Sum the serialized size of its spans, sort descending, and look at the top three. If one payload owns most of the bytes, you have found where to spend your instrumentation effort — and a budget test that will keep it honest.

Replay One Span, Not the Whole Run

Sam Sun — Sun, 18 Oct 2026 07:37:25 +0000

A failed agent run is usually cheaper to diagnose as a single span than as a whole re-run. Pull the span out of the trace, replay it twice, compare fingerprints. That one habit collapses 'it broke again' into a small, decidable answer: the world drifted, the tool is nondeterministic, or the model chose badly while the tool behaved perfectly.

Most trace writing stops at reading spans. Reading is the easy half. Replay is the half that produces evidence you can paste into a bug report and re-run in CI a week later.

Whole-run replay lies to you

A full re-run changes too many variables at once. The working tree moved, the cache warmed, the feature flag flipped, the upstream API returned a slightly different shape. You get a new trace with a new failure and no way to attribute it to any one of those changes.

So stop re-running the run. Re-run the span, where the state is small enough to enumerate.

One span, four fields

The harness below assumes a JSONL trace where each line is a span carrying an id, the tool invocation, the tool result, and the model decision that requested it. If your schema differs, change the accessors, not the idea.

{'span_id': '7f3c', 'tool_call': {'argv': ['git', 'diff', '--stat'], 'env_digest': 'sha256:9a1e', 'timeout_s': 30}, 'tool_result': {'exit_code': 0, 'stdout': ' 3 files changed, 41 insertions(+)', 'stderr': ''}, 'model_decision': {'tool_name': 'git_diff_stat', 'args': {}, 'model_id': '<redacted>'}}

Keep env_digest in the record. Without it, drift analysis is guesswork, because a different PATH or a different tool version reproduces the same argv with different output.

Normalize, then fingerprint

The useful comparison is never raw stdout. Raw stdout carries UUIDs, temp paths, durations, and timestamps that change every run for no diagnostic reason. Normalize first, hash second, and keep the denylist short enough that a reviewer can read it in one screen.

# normalize.py
import hashlib, json, re

VOLATILE = ('ts', 'timestamp', 'started_at', 'ended_at', 'duration_ms', 'trace_id', 'span_id', 'request_id')
UUID = re.compile(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', re.I)
TMP = re.compile(r'/tmp/[A-Za-z0-9_.-]+')

def normalize(obj, drop=VOLATILE):
    if isinstance(obj, dict):
        return {k: normalize(v, drop) for k, v in sorted(obj.items()) if k not in drop}
    if isinstance(obj, list):
        return [normalize(v, drop) for v in obj]
    if isinstance(obj, str):
        return TMP.sub('<tmp>', UUID.sub('<uuid>', obj))
    return obj

def fingerprint(obj):
    blob = json.dumps(normalize(obj), sort_keys=True, separators=(',', ':'))
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

Sixteen hex characters is enough to spot a mismatch and short enough to paste into a commit message. Widen it if you need collision safety across a large corpus; nobody needs that in a debug line.

Two questions, two invocations

Replay has two modes because there are two distinct questions, and mixing them is how people end up blaming the model for a flaky filesystem. Determinism asks whether the tool behaves the same twice. Drift asks whether the recorded result still matches today's world.

# replay.py
import json, subprocess, sys
from normalize import fingerprint

def load(path):
    with open(path) as f:
        return json.load(f)

def run_live(span):
    call = span['tool_call']
    p = subprocess.run(call['argv'], capture_output=True, text=True, timeout=call.get('timeout_s', 30))
    return {'exit_code': p.returncode, 'stdout': p.stdout, 'stderr': p.stderr}

def main(path, mode):
    span = load(path)
    recorded = fingerprint(span['tool_result'])
    if mode == 'determinism':
        first = fingerprint(run_live(span))
        second = fingerprint(run_live(span))
        print(f'recorded={recorded} run1={first} run2={second} stable={first == second}')
    elif mode == 'drift':
        fresh = fingerprint(run_live(span))
        print(f'recorded={recorded} live={fresh} drift={recorded != fresh}')
    else:
        sys.exit(f'unknown mode {mode}')

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])

Usage is a short extraction followed by two commands against the extracted file.

python - <<'PY'
import json
spans = [json.loads(l) for l in open('trace.jsonl') if l.strip()]
json.dump(next(s for s in spans if s['span_id'] == '7f3c'), open('span.json', 'w'), indent=2)
PY
python replay.py span.json determinism
python replay.py span.json drift

Run determinism first. Replaying a nondeterministic tool against a recording tells you nothing except that clocks exist.

Reading the two booleans

Each mode answers yes or no, and the pair is more informative than either alone. The top-left cell is the one people skip, and it is the most common real answer in my experience of these loops.

stable (run1 == run2)	drift (recorded == live)	reading	next move
yes	yes	tool and world unchanged	the model's plan was wrong; replay the decision
yes	no	input or environment changed	diff env_digest and pinned inputs first
no	no	tool is nondeterministic	capture a distribution, not one fixture
no	yes	rare branch or coincidence	repeat determinism ten times before concluding

When the tool is stable and the world has not moved, the failure lives in the decision layer, and no amount of tool debugging will find it. That is where a second model earns its keep.

Replay the decision with a second model

Take the recorded prompt and tool schema for that span, ask a different model for the tool call, and compare the emitted call's fingerprint to the recorded one. A different call means the decision is model-dependent. An identical call means the decision is robust and the bug sits downstream. Keep the schema byte-identical, or you are testing the schema.

This is a one-call, cheap check, which makes it a reasonable fit for a free model tier rather than a paid production lane. MonkeyCode is where I run that second-opinion step: its free model access covers the occasional decision replay, and its free server option is enough to host the replay worker itself. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The worker shape matters more than the vendor. Replay is bursty, I/O-bound, and needs no SLA: it runs when a trace fails and idles the rest of the week. Putting that worker on the same host as the production runner is how a debugging tool takes down a service. Keep the lanes separate and let the replay lane be disposable. If you point a client at a model endpoint, read the base URL and model ids from current docs rather than from a blog post, and keep credentials in an environment variable.

Limitations, stated plainly

Span replay cannot see side effects that already happened. If the tool wrote to a database or opened a pull request, replaying it live will write again; stub those tools and compare the stub's arguments instead of the result. Redact before you store, because a trace file is a data-retention decision you make once and regret for years.

Free tiers are not capacity plans. A free token allowance and a free server option are fine for replaying tens or hundreds of spans, and they are the wrong foundation for an audit archive or a latency-sensitive loop. Treat quotas as operator-supplied and verify current terms yourself; they change without notice.

Skip this approach if your artifacts cannot leave your environment, if your tools cannot be stubbed, or if your failures are dominated by concurrent state rather than single-span behaviour. Single-span replay is a narrow instrument, and it is sharp only in that narrowness.

For everyone else the loop is two commands wide. A trace fails, you extract the span, you run determinism, you run drift, and you have a defensible answer before you touch a prompt. If you want to try the second-opinion step without provisioning anything, the free token allowance and the free server option are enough to start; confirm the current terms first.

Count Tokens per Span, Not per Run

Sam Sun — Sat, 17 Oct 2026 06:35:28 +0000

Count Tokens per Span, Not per Run

A run total is an invoice. It tells you what you owe, and nothing about what to change. The moment an agent run costs more than you expected, the useful question is which span spent the tokens: the retrieval call, the tool schema you re-sent twelve times, the retry loop, the summarizer that quietly grew. One integer per run cannot answer that. A dashboard that refreshes hourly answers it after you have already burned the budget.

So the artifact here is small and boring. An append-only ledger records usage per span. A reconciler compares your local estimate against the provider's declared usage. A breaker refuses the next call once the run has crossed its cap. Standard library only, no tracing vendor, and no rewritten lines — which matters more than it sounds, because a crash at step nine of a twenty-step run should not cost you the first eight records.

This week's DEV front page swung between AI fatigue and open-source AI project pushes. Both are symptoms of the same gap: the tooling got loud, the accounting stayed quiet. Token spend is where that gap is cheapest to close.

The number you can't act on

Run-level totals conflate three different costs. There is the fixed cost of your prompt scaffolding, which you pay on every call whether or not the model needs it. There is the variable cost of tool output, which is where long JSON blobs and file reads land. And there is the cost of repetition: retries, re-plans, and the third attempt at the same failing edit.

A single total blends all three into a number that changes when any of them changes. You end up optimizing the scaffolding because it is the part you can see in your source, while the retry loop quietly doubles the bill. The ledger below exists so you can check that hypothesis against your own traffic instead of taking anyone's word for it — mine included.

A ledger that answers in spans

Write the record before you need it. Each span appends one open line and one closed line; nothing is ever mutated in place, so a killed process leaves a parseable prefix rather than a corrupt file.

# spend.py — append-only per-span token ledger with a budget breaker
import json, pathlib, time, uuid


class BudgetExceeded(RuntimeError):
    """Raised after the overshooting span is recorded, before the next call."""


class SpanLedger:
    def __init__(self, path="runs/ledger.jsonl", cap_tokens=200_000, reserve_tokens=2_000):
        self.path = pathlib.Path(path)
        self.path.parent.mkdir(parents=True, exist_ok=True)
        self.cap = cap_tokens
        self.reserve = reserve_tokens
        self.spent = 0
        self.run_id = uuid.uuid4().hex[:12]

    def _append(self, rec):
        line = json.dumps(rec, ensure_ascii=False, sort_keys=True)
        with self.path.open("a", encoding="utf-8") as f:
            f.write(line + "\n")

    def open(self, name, parent=None):
        self._append({"run_id": self.run_id, "span": name, "parent": parent,
                      "started_at": round(time.time(), 3), "status": "open"})
        return name

    def close(self, name, parent=None, usage=None, estimate_total=None):
        prompt = int((usage or {}).get("prompt_tokens", 0))
        completion = int((usage or {}).get("completion_tokens", 0))
        total = prompt + completion
        self.spent += total
        rec = {"run_id": self.run_id, "span": name, "parent": parent,
               "status": "closed", "prompt_tokens": prompt,
               "completion_tokens": completion, "total_tokens": total,
               "estimate_total": estimate_total,
               "drift": None if estimate_total is None else total - estimate_total,
               "cumulative_tokens": self.spent,
               "ended_at": round(time.time(), 3)}
        self._append(rec)
        return rec

    def guard(self):
        if self.spent + self.reserve > self.cap:
            raise BudgetExceeded(
                f"spent={self.spent} reserve={self.reserve} cap={self.cap}")

The ordering in close() is the whole design. The record is durably appended first, then guard() is called separately by the driver. That way the span which pushed you over budget is still in the file — you want to see it — and the refusal applies to the next call, which has not happened yet.

The reserve is not decoration. A breaker can only act on usage already reported, so it cannot stop the call in flight. You have to reserve headroom at least as large as your largest expected completion, otherwise the cap is a target rather than a limit.

ledger = SpanLedger(cap_tokens=120_000, reserve_tokens=4_000)
try:
    for step, prompt in enumerate(workload):
        span = f"step[{step}]"
        ledger.open(span, parent="run")
        reply = provider(prompt)              # must expose a usage payload
        ledger.close(span, parent="run", usage=reply.get("usage"),
                     estimate_total=estimate(prompt, reply.get("text", "")))
        ledger.guard()
        apply(reply)
except BudgetExceeded as e:
    log.warning("run halted: %s", e)

Reconcile the estimate against the declared usage

Every local tokenizer is wrong about someone's traffic. Tool schemas, minified JSON, and non-English text are the usual disagreements, because your estimator and the provider's tokenizer segment them differently. Store both numbers per span and the disagreement becomes a measurement rather than a mystery.

def reconcile(path, tolerance=0.05, top=5):
    rows = [json.loads(l) for l in pathlib.Path(path).read_text().splitlines() if l.strip()]
    closed = [r for r in rows if r.get("status") == "closed" and r.get("total_tokens")]
    flagged, abs_drift = [], 0
    for r in closed:
        est = r.get("estimate_total")
        if not est:
            continue
        rel = (r["total_tokens"] - est) / est
        abs_drift += abs(r["total_tokens"] - est)
        if abs(rel) > tolerance:
            flagged.append((r["span"], r["total_tokens"], est, round(rel, 3)))
    return {"spans": len(closed), "abs_drift_tokens": abs_drift,
            "flagged": sorted(flagged, key=lambda t: -abs(t[3]))[:top]}

Run it as python -m spend reconcile runs/ledger.jsonl --tolerance 0.05. The output is not a score. It is a list of spans whose cost you currently cannot predict, which is exactly the list to look at before you tune anything.

A three-check test plan you can run offline

The self-test below needs no network and no API key, because the provider is a stub. Treat the assertions as a specification of the behavior you want; expect to adjust the token arithmetic once you see your real provider's usage payload.

# selftest.py
import json, pathlib, tempfile, spend

calls = {"n": 0}

def fake_provider(prompt, completion_tokens=40):
    calls["n"] += 1
    return {"text": "x" * 10,
            "usage": {"prompt_tokens": len(prompt.split()),
                      "completion_tokens": completion_tokens}}

def estimate(prompt, text):
    return len(prompt.split()) + max(1, len(text) // 4)   # deliberately naive

P = "a b c d e f g h"
tmp = pathlib.Path(tempfile.mkdtemp())
ledger = spend.SpanLedger(tmp / "l.jsonl", cap_tokens=100, reserve_tokens=0)

ran = 0
try:
    for i in range(50):
        ledger.open(f"s{i}")
        r = fake_provider(P)
        ledger.close(f"s{i}", usage=r["usage"], estimate_total=estimate(P, r["text"]))
        ledger.guard()
        ran += 1
except spend.BudgetExceeded:
    pass

rows = [json.loads(l) for l in (tmp / "l.jsonl").read_text().splitlines()]
assert calls["n"] == ran + 1            # overshoot recorded, next call refused
assert sum(r.get("total_tokens", 0) for r in rows) <= 200
assert all(r["status"] in {"open", "closed"} for r in rows)

Three properties are worth checking before you trust the harness on real traffic. Does the breaker stop the next call rather than the current one? Does the sum of recorded spans match what you would have been billed? And does a SIGKILL mid-run leave a file you can still sum? The third one you can verify by hand in a minute: kill the process, then wc -l runs/ledger.jsonl and parse whatever is there.

Gate	Pass condition	What failure means
Breaker ordering	Overshooting span present, next call absent	You are capping after the fact
Reconciliation	Flagged spans are a short list	Estimator is too coarse to tune against
Crash safety	Truncated tail parses line-by-line	Something rewrites instead of appends

Where a free hosted run fits

None of this requires infrastructure — a JSONL file and a loop. What it does require is repetition: you need several comparable runs before the drift list means anything, and that is the point where local inference or a paid key starts to cost real money for what is essentially a measurement exercise.

MonkeyCode, an open-source coding agent project, is relevant here for a narrow reason: the operator states that it offers free model access and a free server option, with a free allowance reported as ten million tokens. I have not independently verified those terms, and allowances of this kind change. If you use that route, run the ledger harness against two or three runs and let the numbers decide whether it fits your workload; check the project's current terms before you plan around any figure. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limits, and who should skip this

The ledger is per-provider by construction. Token counts are tokenizer-specific, so summing spans across two providers gives you a number with no unit. Streamed responses may report usage late or not at all, in which case the span lands with total_tokens: 0 and your reconciliation silently skips it — worth an explicit assertion in production.

Skip this if you already reconcile a provider-side cost dashboard against invoices, or if you need per-tenant chargeback with currency and rounding rules; that is billing, and billing wants a real telemetry pipeline rather than a file you append to. Skip it too if your runs are single-call scripts — the run total and the span total are the same number, and you have gained a class and a file handle for nothing.

For everyone else the discipline is the deliverable. Record the span before you tune the prompt, reconcile the estimate before you trust it, and refuse the next call while refusal is still cheap.

Two Clocks, One Agent Trace

Sam Sun — Fri, 16 Oct 2026 05:41:21 +0000

Two Clocks, One Agent Trace

If a span record carries both a wall-clock timestamp and a monotonic one, assume they will disagree on the run you actually care about. A single NTP step of two seconds is enough to invert the completion order of two tool calls, and from that moment your timeline viewer is rendering fiction. You then debug the fiction.

This is not an exotic edge case. Wall clocks get stepped when a host corrects accumulated drift, when a VM resumes after suspend, and when two containers in the same pod end up in different clock namespaces. A monotonic clock keeps counting through all of it, at the cost of being meaningless across machines. Traces need both numbers, and a debug loop needs to say out loud which one it is currently trusting.

The failure is quiet, and it looks like an agent bug

An agent run is a tree: a turn span opens, tool spans nest underneath it, and the interesting part is the order in which the tools actually executed. Almost every trace viewer sorts that tree by a timestamp. Sort on wall-clock start time and the tree looks plausible most of the time, which is exactly why the bug survives review.

Inject a step and the same tree contradicts itself. A tool.http_fetch span reports a wall duration of 2.5 seconds while its own monotonic delta says 0.5 seconds, because the clock moved forward while the span was open. A sibling tool.repo_diff then appears to finish before the fetch that preceded it. The tempting conclusion is that the agent issued calls out of order; the correct conclusion is that one column of your trace is arithmetic on a quantity that is not monotonic.

The fix is not to throw away wall clock. Humans need it, log correlation needs it, and your incident timeline needs it. The fix is to derive ordering from structure and a monotonic source, and to use wall clock only for display once the trace has been audited.

A span record that can be audited

Keep the record boring and explicit. One span per JSONL line, with a parent link, a per-process sequence counter, and both clocks sampled at the same two points in the span's life.

{"span_id":"s1","parent_id":"s0","tool":"tool.http_fetch","seq":1,
 "wall_start_ns":1700000000100000000,"wall_end_ns":1700000002500000000,
 "mono_start_ns":8200000000,"mono_end_ns":8500000000}

The seq field is the underrated one. It is a plain integer counter incremented in the process that opens the span, so it stays correct through any clock event, any suspend, and any migration. It cannot order work across processes, and it is not a distributed clock. Inside one process it is the most trustworthy ordering signal you have, and it costs one integer per span.

The auditor

The script below is standard library only, reads JSONL, and emits a report plus a reconstructed order. It checks three things that can each be verified locally: monotonic duration must not be negative, wall-minus-monotonic drift per span must stay under a tolerance, and a child's wall window must not escape its parent's when the child's own drift is small.

# clock_audit.py - Python 3.10+, standard library only.
import json
import sys
from dataclasses import dataclass

TOLERANCE_NS = 50_000_000  # 50 ms; above this, treat drift as a clock step


@dataclass(frozen=True)
class Span:
    span_id: str
    parent_id: str | None
    tool: str
    seq: int
    wall_start_ns: int
    wall_end_ns: int
    mono_start_ns: int
    mono_end_ns: int


def read_spans(path):
    spans = []
    required = {"span_id", "tool", "seq", "wall_start_ns",
                "wall_end_ns", "mono_start_ns", "mono_end_ns"}
    with open(path, encoding="utf-8") as fh:
        for lineno, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                row = json.loads(line)
            except json.JSONDecodeError as exc:
                raise SystemExit(f"{path}:{lineno}: bad JSON: {exc}") from exc
            missing = required - row.keys()
            if missing:
                raise SystemExit(f"{path}:{lineno}: missing {sorted(missing)}")
            spans.append(Span(row["span_id"], row.get("parent_id"), row["tool"],
                              row["seq"], row["wall_start_ns"], row["wall_end_ns"],
                              row["mono_start_ns"], row["mono_end_ns"]))
    if not spans:
        raise SystemExit(f"{path}: no spans")
    return spans


def audit(spans):
    by_id = {s.span_id: s for s in spans}
    problems = []
    for s in spans:
        wall = s.wall_end_ns - s.wall_start_ns
        mono = s.mono_end_ns - s.mono_start_ns
        drift = wall - mono
        if mono < 0:
            problems.append({"kind": "mono_inverted", "span": s.span_id,
                             "detail": f"monotonic duration {mono} ns"})
        if abs(drift) > TOLERANCE_NS:
            problems.append({"kind": "clock_step", "span": s.span_id,
                             "detail": f"wall-mono drift {drift / 1e6:.1f} ms"})
        parent = by_id.get(s.parent_id) if s.parent_id else None
        if parent and abs(drift) <= TOLERANCE_NS:
            if s.wall_start_ns < parent.wall_start_ns or s.wall_end_ns > parent.wall_end_ns:
                problems.append({"kind": "parent_violation", "span": s.span_id,
                                 "parent": parent.span_id,
                                 "detail": "child window escapes parent"})
    return {"spans": len(spans), "problems": problems,
            "timeline_trusted": not problems}


def causal_order(spans):
    """Order by parent links, then by (per-process seq, mono_start)."""
    by_id = {s.span_id: s for s in spans}
    kids, roots = {}, []
    for s in spans:
        if s.parent_id and s.parent_id in by_id:
            kids.setdefault(s.parent_id, []).append(s)
        else:
            roots.append(s)

    key = lambda s: (s.seq, s.mono_start_ns)
    ordered = []

    def walk(node):
        ordered.append(node)
        for child in sorted(kids.get(node.span_id, []), key=key):
            walk(child)

    for root in sorted(roots, key=key):
        walk(root)
    return ordered


if __name__ == "__main__":
    spans = read_spans(sys.argv[1])
    report = audit(spans)
    report["order"] = [s.span_id for s in causal_order(spans)]
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    sys.exit(0 if report["timeline_trusted"] else 2)

The non-zero exit code matters more than the report. Wire the auditor into the same place that regenerates your trace fixtures, and an untrusted timeline fails a job instead of quietly appearing in a dashboard.

Reproduce the step without touching a real clock

Manipulating a live clock is possible, but it is the wrong first move: libfaketime works through LD_PRELOAD and the exact invocation depends on your distro's library path and time format, while date -s inside a container needs privileges most CI runners will not grant. Inject the step into a fixture instead. It is deterministic, it runs anywhere, and it pins the exact failure you are trying to catch.

# test_clock_audit.py - run with: pytest -q test_clock_audit.py
import json, pathlib, subprocess, sys

STEP_NS = 2_000_000_000  # one 2 s NTP step


def write_fixture(tmp: pathlib.Path) -> pathlib.Path:
    w, m = 1_700_000_000_000_000_000, 8_000_000_000
    spans = [
        dict(span_id="s0", parent_id=None, tool="agent.turn", seq=0,
             wall_start_ns=w, wall_end_ns=w + 900_000_000,
             mono_start_ns=m, mono_end_ns=m + 900_000_000),
        # the clock moves forward while this span is open
        dict(span_id="s1", parent_id="s0", tool="tool.http_fetch", seq=1,
             wall_start_ns=w + 100_000_000, wall_end_ns=w + 500_000_000 + STEP_NS,
             mono_start_ns=m + 100_000_000, mono_end_ns=m + 500_000_000),
        dict(span_id="s2", parent_id="s0", tool="tool.repo_diff", seq=2,
             wall_start_ns=w + 600_000_000, wall_end_ns=w + 800_000_000,
             mono_start_ns=m + 600_000_000, mono_end_ns=m + 800_000_000),
    ]
    path = tmp / "trace.jsonl"
    path.write_text("\n".join(json.dumps(s) for s in spans) + "\n", encoding="utf-8")
    return path


def test_step_is_reported_and_order_survives(tmp_path):
    trace = write_fixture(tmp_path)
    out = subprocess.run([sys.executable, "clock_audit.py", str(trace)],
                         capture_output=True, text=True)
    report = json.loads(out.stdout)
    assert report["timeline_trusted"] is False
    assert out.returncode == 2
    assert "clock_step" in {p["kind"] for p in report["problems"]}
    assert report["order"] == ["s0", "s1", "s2"]

The fixture encodes both halves of the lesson. Sorting by wall end time puts s2 before s1, because s1 absorbed the step; the structural order built from parent_id and seq keeps the true sequence. Run the auditor directly and the report says so:

$ python clock_audit.py trace.jsonl
{
  "spans": 3,
  "problems": [
    {
      "kind": "clock_step",
      "span": "s1",
      "detail": "wall-mono drift 2000.0 ms"
    }
  ],
  "timeline_trusted": false,
  "order": ["s0", "s1", "s2"]
}
$ echo $?
2

Read the report as triage, not as a verdict

Symptom	Likely cause	Cheap check	Action
Wall duration far exceeds monotonic duration	NTP step, VM suspend, or clock namespace swap	per-span drift	Trust monotonic; mark the wall column suspect in that window
Child window escapes parent while drift is small	Async callback recorded under the wrong parent	`seq` gaps between siblings	Fix span parenting, not the timestamps
Monotonic duration is negative	Process re-exec, or two clock domains merged into one file	process or PID tag on the span	Split the trace per clock domain before ordering
Clocks agree and order is still wrong	Genuine concurrency with no causal link	concurrent siblings with equal parents	Add a span link or an explicit causal edge

That last row is the honest boundary of this method. Two parallel tool calls with no parent-child relationship are simply concurrent, and a flat sort will always impose an order that the system never had.

Where a free tier actually fits

Generating skewed traces by hand gets old after about the fourth variant, and the useful corpus is the boring one: hundreds of malformed traces covering steps at different points inside a span. I used MonkeyCode's free model access to generate those variants and its free server option to run the auditor and the fixture suite off my own machine, so the job did not depend on a laptop staying awake. As of this writing the operator advertises a 10M-token allowance on the free tier plus the free server option; allowances and hardware change, so check the current pricing page before you plan around either number.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Nothing in the auditor depends on that tier. The script, the fixture, and the pytest file above run on any Python 3.10+ interpreter, and if you already have a trace source you trust, you can point the auditor at a sample today and see whether the timeline you have been reading is the one that happened.

Limits, and who should skip this

This auditor assumes roughly one process per trace file and a sequence counter that is consistent within it. Cross-host tool calls need vector clocks or an explicit causal edge, not a per-process integer, and pretending otherwise will produce confident nonsense. It also flags suspects; it does not restore correctness. If you need billing-grade timing, NTP monitoring and synchronized hosts are the real fix, and this script is at best the alarm that tells you to go look.

Skip the whole approach if your SDK already emits monotonic timestamps with stable parent links and you trust the collector that assembles them. Adding a second auditor to a pipeline that is already correct is duplicate work, and a second source of ordering truth is worse than one. Skip it too if you cannot change span emission at all: this tool can tell you a wall window is unreliable, but it cannot recover an order that was never recorded.

Agent Traces Hide Their Gaps: Build a Span-Gap Ledger

Sam Sun — Thu, 15 Oct 2026 11:39:28 +0000

Agent Traces Hide Their Gaps: Build a Span-Gap Ledger

A trace that draws every span can still fail to explain a run. Take the wall clock of the root span, subtract the exclusive time of every span beneath it, and whatever remains is the part of the run nobody instrumented. That residual is usually where the bug lives, and no waterfall view will point at it, because a waterfall only draws what it was given.

Take a synthetic turn to make the arithmetic concrete: a root span of 1840 ms, one model call of 190 ms, two tool calls of 8 ms and 2 ms. Roughly 1.6 seconds are unaccounted for. That time did not vanish. It went into a control loop that polls, an HTTP client redoing TLS, a retry sleeping before it re-issues a request, or a serializer walking a large payload. None of those emit spans by default, so none of them appear.

Exclusive time is the concept the ledger is built on. For any span, exclusive time is its duration minus the union of its direct children's intervals. A parent that waits 300 ms while a child runs 120 ms has 180 ms of exclusive time, and that 180 ms is a mixture of real work and empty waiting. The ledger separates the two by listing the covered intervals and the holes between them.

The invariant worth asserting

For a strictly nested trace, where every child sits inside its parent and siblings never overlap, exclusive time sums to the root duration exactly:

wall_clock == sum over all spans s of (duration(s) - union(direct children of s))

The identity telescopes: every non-root span is subtracted once from its parent and added once on its own line. A residual under this assumption is not rounding noise, it is a topology defect. Overlapping siblings, a child whose interval escapes its parent, or timestamps taken from two unsynchronized clocks all break the identity, and each of them means your trace is lying to you before you even start interpreting it.

That is why the ledger prints the residual next to the wall clock instead of burying it. If you run tools in parallel on purpose, the residual is the overlap you asked for and you should compare it against the overlap you expect. If you did not, stop and fix the topology first.

The ledger, in one file

The script below has no dependencies outside the standard library. It reads an OTLP/JSON export or a flat JSON list of spans, builds the parent/child index, and emits one row per span and one row per gap.

#!/usr/bin/env python3
"""span_gap_ledger.py - account for every millisecond of one agent run.

Input: OTLP/JSON (resourceSpans -> scopeSpans -> spans) or a flat JSON list.
Output: exclusive span time plus empty stretches inside each span.
"""
import json
import sys
from dataclasses import dataclass


@dataclass
class Span:
    sid: str
    parent: str
    name: str
    start: int  # ns
    end: int    # ns

    @property
    def dur(self) -> int:
        return self.end - self.start


def load(path):
    doc = json.load(open(path))
    if isinstance(doc, list):
        blocks = [{"scopeSpans": [{"spans": doc}]}]
    else:
        blocks = doc.get("resourceSpans", [doc])
    spans = []
    for rs in blocks:
        scopes = rs.get("scopeSpans") or rs.get("instrumentationLibrarySpans") or []
        for ss in scopes:
            for s in ss.get("spans", []):
                spans.append(Span(s["spanId"], s.get("parentSpanId") or "",
                                  s["name"], int(s["startTimeUnixNano"]),
                                  int(s["endTimeUnixNano"])))
    return spans


def index(spans):
    children = {s.sid: [] for s in spans}
    for s in spans:
        children.setdefault(s.parent, []).append(s)
    roots = children.get("", [])
    if not roots:
        raise SystemExit("no root span found; export the full trace")
    return children, max(roots, key=lambda s: s.dur)


def merge(intervals):
    out = []
    for a, b in sorted(intervals):
        if out and a <= out[-1][1]:
            out[-1][1] = max(out[-1][1], b)
        else:
            out.append([a, b])
    return out


def ledger(root, children):
    rows = []

    def visit(span, depth):
        clipped = []
        for c in children.get(span.sid, ()):
            a, b = max(c.start, span.start), min(c.end, span.end)
            if b > a:
                clipped.append((a, b))
        covered = merge(clipped)
        own = span.dur - sum(b - a for a, b in covered)
        rows.append({"kind": "span", "owner": span.name, "depth": depth,
                     "offset_ms": (span.start - root.start) / 1e6,
                     "ms": own / 1e6})
        cursor = span.start
        for a, b in covered:
            if a > cursor:
                rows.append({"kind": "gap", "owner": span.name, "depth": depth,
                             "offset_ms": (cursor - root.start) / 1e6,
                             "ms": (a - cursor) / 1e6})
            cursor = max(cursor, b)
        if span.end > cursor:
            rows.append({"kind": "gap", "owner": span.name, "depth": depth,
                         "offset_ms": (cursor - root.start) / 1e6,
                         "ms": (span.end - cursor) / 1e6})
        for c in sorted(children.get(span.sid, ()), key=lambda s: s.start):
            visit(c, depth + 1)

    visit(root, 0)
    return rows

The driver and a self-test that proves the identity without needing an agent:

def self_test():
    ms = 1_000_000
    spans = [Span("r", "", "agent.run", 0, 1000 * ms),
             Span("c", "r", "llm.plan", 100 * ms, 400 * ms),
             Span("g", "c", "tool.http_fetch", 150 * ms, 250 * ms)]
    children = {s.sid: [] for s in spans}
    for s in spans:
        children.setdefault(s.parent, []).append(s)
    rows = ledger(spans[0], children)
    total = sum(r["ms"] for r in rows if r["kind"] == "span")
    assert abs(total - 1000.0) < 1e-9, total
    print(f"invariant holds: {total:.1f} ms exclusive == 1000.0 ms wall clock")


def main(argv):
    if len(argv) > 1 and argv[1] == "--self-test":
        return self_test()
    spans = load(argv[1])
    children, root = index(spans)
    rows = ledger(root, children)
    own = sum(r["ms"] for r in rows if r["kind"] == "span")
    wall = root.dur / 1e6
    print(f"wall clock   {wall:9.1f} ms")
    print(f"exclusive S  {own:9.1f} ms")
    print(f"residual     {wall - own:9.1f} ms  (overlap, escaping spans, or clock skew)")
    print("\ntop gaps")
    gaps = sorted((r for r in rows if r["kind"] == "gap"), key=lambda r: -r["ms"])
    for r in gaps[:8]:
        print(f"  {r['ms']:8.1f} ms  t+{r['offset_ms']:8.1f}  "
              f"{'  ' * r['depth']}{r['owner']}")


if __name__ == "__main__":
    main(sys.argv)

Run it against a real export with python span_gap_ledger.py trace.json, or against the built-in fixture with python span_gap_ledger.py --self-test. The rows are plain dicts, so feeding the CI check below is one json.dump away.

Reading the top of the list

Sorted descending, the gaps form a budget you can act on. A synthetic example, not a measurement of any product:

     412.3 ms  t+  211.0  agent.run
      96.1 ms  t+  520.4    tool.shell_exec
      31.7 ms  t+ 1188.9    llm.plan
       8.4 ms  t+ 1402.2    tool.http_fetch
       0.3 ms  t+   12.9  agent.run

Each row names the span whose exclusive time contains the hole, and that owner tells you which layer to look at. A gap inside agent.run is the orchestrator itself: loop bookkeeping, prompt assembly, JSON serialization of a large tool result. A gap inside llm.plan is the provider client: retry backoff, connection setup, token counting done locally. A gap inside tool.* is the tool's own runtime, which often means a subprocess or an HTTP client doing work the tool never bracketed.

The decision rule for instrumentation stays narrow. Add a span only when a gap corresponds to a named unit of work you would want to debug on its own, and leave pure latency alone. Wrapping every wait in a span inflates the trace, slows the collector, and gives you a list of spans named sleep that nobody reads. One gap of 412 ms with a clear owner beats forty spans that restate it.

Turning the ledger into a regression check

Gap analysis is most useful as a diff between two runs of the same scenario. Compare the largest gap before and after a change; if it moved to a different owner, the change relocated the latency rather than removing it.

# ci_gap_diff.py - fail when the biggest gap moves or grows
import json
import sys


def top_gap(path):
    rows = json.load(open(path))
    gaps = [r for r in rows if r["kind"] == "gap"]
    return max(gaps, key=lambda r: r["ms"], default={"owner": None, "ms": 0.0})


before, after = top_gap(sys.argv[1]), top_gap(sys.argv[2])
drift = after["ms"] - before["ms"]
if after["owner"] != before["owner"] or drift > 250:  # a policy, not a law
    print(f"top gap moved: {before['owner']} {before['ms']:.1f}ms -> "
          f"{after['owner']} {after['ms']:.1f}ms")
    sys.exit(1)

The 250 ms threshold is a starting policy tuned to the noise floor of your environment, not a universal constant. On a quiet machine, single-process, sub-millisecond gaps are scheduling jitter and should be filtered out entirely.

Where a free tier fits the loop

Reproducing a run costs tokens, and the ledger step costs almost nothing because it is pure integer arithmetic over a JSON file. That split matters for anyone rebuilding a debug loop on a side project. MonkeyCode offers free model access and a free server option (availability stated by the operator), which makes the expensive half of the loop — re-running the agent to capture a fresh trace — cheap enough to do repeatedly, while the ledger runs wherever your agent already writes its spans.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to test the loop end to end, pick one failing run, generate a second trace after a fix, and check whether the top gap moved owners. That single comparison tells you more than another hour of reading span names.

What the ledger cannot tell you

It measures time, not correctness. A fully accounted run can still be wrong, and a run with a huge gap can be perfectly healthy because the gap is a third-party API you cannot instrument.

Timestamps are the weak point. Nanosecond fields written by different processes, containers, or hosts come from different clocks, and unsynchronized clocks manufacture gaps that never happened. Keep ledger comparisons within a single process with a monotonic clock until you have verified sync across the boundary.

Sampling invalidates everything. If the run was sampled below 100%, the ledger describes the sample, not the run, and the totals are fiction. Force full capture on the specific run you are debugging. Suspended runs are a second trap: an agent paused for human approval produces one giant gap that is neither a bug nor missing instrumentation, so annotate or exclude it before sorting.

Finally, some teams should skip this. If you already run a mature APM with per-service latency budgets and alerting, the ledger is a debugging tool, not a replacement, and it will mostly restate what you have. If your workload is a single-threaded script with no spans, a profiler answers the same question in one command. The ledger earns its keep in the middle: multi-step agent runs where the wall clock and the span tree disagree, and nobody knows which one to believe.

Close the Trace on a Matching Receipt

Sam Sun — Tue, 13 Oct 2026 03:26:44 +0000

An agent trace is closed when a local export receipt names the same tool set as the plan. A zero exit code does not close it. Process exit only proves the runtime stopped, which is a weaker fact than it looks.

The loading dock is the useful analogy. A truck door can shut while two crates remain on the pallet. The exit code is that door. The receipt is the scan of every crate ID.

When those lists disagree, a remote trace UI is a tour of an incomplete shipment. The missing crate gets blamed on a warehouse that never received it. That misread wastes the next edit.

Agent runs widen the gap. A model writes a plan, a runner invokes tools, and an exporter flushes spans on a timer the process does not share. Those clocks drift.

A tool can return, the process can exit, and the last batch can still sit in a buffer. A query at that moment reports a missing span. The defect is the gate, not the tool.

Membership before meaning

Three sets matter, and only two of them decide this gate. The plan is the inventory of tool names the run was supposed to invoke, stored as JSON before execution. The emitted set is what the local tracer recorded, failures included. The exported set is what a file or a collector acknowledged after flush.

Inspection waits until the plan and the exported set are equal for one run id. Emitted but not exported is an exporter problem. Planned but not emitted is a runner problem.

Exported but not planned is an unexpected tool, or a stale span that leaked across runs. Those are different edits. A combined log makes them look like one mysterious failure.

Name equality does not mean the run was correct. A span can carry the right name and a wrong argument, a partial write, or an error status. Equality answers a prior question only: is the trace complete enough to inspect?

Argument hashes and status checks are a later pass. Folding them into the receipt hides whether you lack data or already hold bad data. Separate files keep those failures distinct.

Parent and child linkage is a different question. This gate does not rebuild causality, and it does not score a retry. It checks membership. If the name is absent, there is nothing to link and nothing worth retrying.

Keep the first judge on disk

The comparison belongs next to the process, before any network hop. A remote collector, including a free one, is a second shelf. It is a poor first judge.

Network delay and flush delay produce the same query result: the span is not there. A dashboard cannot tell those delays apart. It will send you to edit the runner when the exporter simply has not finished.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. After the receipt is closed and the bodies are redacted, MonkeyCode's free model access can lint a plan file against a tool catalog you already trust, and its free server option can hold the exported trace for a later read. That is optional convenience, not the decision.

This article states no token allotment, model name, retention window, or hardware shape, because those facts go stale. Read the current docs before you depend on either option. Delete the product references and the local gate still stands.

Do not let the model invent the expected set. A model can skip a tool and then confirm a plan that matches its own skip. Pin the catalog in the repo. Let a model fill arguments if you want, but keep membership under version control when coverage is the question you are asking.

A receipt you can rerun

The script below is a proposal. It is not a recorded benchmark, and it does not open a socket. It writes receipt.json and exits 2 when the sets differ.

#!/usr/bin/env python3
"""Local receipt gate. Unexecuted example, not a measured result."""
import json
import sys
from pathlib import Path

def load_names(path, key, run_id):
    rows = json.loads(Path(path).read_text(encoding="utf-8"))
    found = []
    for row in rows:
        if row.get("run_id") != run_id:
            continue
        found.append(row[key])
    return set(found)

def main():
    run_id = sys.argv[3]
    planned = load_names(sys.argv[1], "tool", run_id)
    exported = load_names(sys.argv[2], "name", run_id)
    missing = sorted(planned - exported)
    extra = sorted(exported - planned)
    receipt = {
        "run_id": run_id,
        "planned_count": len(planned),
        "exported_count": len(exported),
        "missing": missing,
        "extra": extra,
        "closed": (not missing) and (not extra),
    }
    Path("receipt.json").write_text(
        json.dumps(receipt, indent=2) + "\n", encoding="utf-8"
    )
    print(json.dumps(receipt))
    return 0 if receipt["closed"] else 2

if __name__ == "__main__":
    raise SystemExit(main())

Two fixtures make the check reproducible without a live agent. Save the plan as plan.json.

[
  {"run_id": "run-17", "tool": "read_file"},
  {"run_id": "run-17", "tool": "apply_patch"}
]

Save the flushed export as spans.json.

[
  {"run_id": "run-17", "name": "read_file", "status": "ok"},
  {"run_id": "run-17", "name": "apply_patch", "status": "ok"}
]

python3 receipt_gate.py plan.json spans.json run-17
echo "gate_exit=$?"
python3 -c 'import json; print(json.load(open("receipt.json"))["closed"])'

Exit 0 means the sets match for run-17. Exit 2 means stop. Do not open a remote query, and do not ask a model to explain a span that never left the machine.

The run id is the isolation boundary. Without it, a previous run's extra name fails a good plan, and you debug a ghost. Filter before you subtract.

If the exporter writes OTLP JSON, reduce it before the gate. This jq path is a schema assumption, not a standard. Adjust it to your exporter, or the gate will report every tool as missing, which is a signal about the adapter rather than the agent.

jq '[.resourceSpans[].scopeSpans[].spans[] | {run_id: (.attributes["run.id"] // .attributes["run_id"]), name: .name, status: (.status.code // "unset")}]' otel.json > spans.json

A short numeric walk-through, still hypothetical, shows why the counts are enough. Suppose the plan lists four tools and the export lists three. missing has length 1. That name is the next action.

You still need one local split before you edit anything. Count the name in the emitted log, then count it in the export file. The two integers name the owner.

grep -c '"tool": "apply_patch"' emitted.jsonl
grep -c '"name": "apply_patch"' spans.json

If the emitted count is already zero, the runner skipped the call. If the emitted count is one and the export count is zero, the flush did not finish. Same missing name, opposite owner.

The receipt does not make that split by itself. The two counts do, and they stay on disk where you can rerun them. Keep both commands next to the script so the split is repeatable.

Keep the fixtures in the same commit as the script. A gate without pinned inputs becomes a story about yesterday's exporter. When the schema moves, the fixtures fail in CI before anyone spends an afternoon on a blank trace. That failure is the point: planned_count and exported_count diverged, and the run id is quotable.

Difference	Local signal	Next edit
Missing name	Absent in the emitted log	Runner skip or catalog drift
Missing name	Present in the emitted log, absent after export	Flush or exporter
Extra name	Exported count above the plan	Run id leak or unexpected tool
Empty differences	Counts equal and `closed` is true	Arguments and status, not membership

Read the table as a routing rule, not as a score. One row should match. If two rows match, the run id filter failed and the sets are mixed. Fix isolation before you interpret either difference.

What not to conclude

Treat a green receipt as permission to inspect, not as a health certificate. After closed is true, compare argument hashes and refuse a healthy label when any status is error. Keep that pass in a second file so a red status cannot be mistaken for a missing span.

Redact before any upload. Tool arguments carry paths, tokens, and customer text. A free server does not repair a redaction miss. If you cannot strip those fields with confidence, leave the receipt local and skip the remote shelf.

Skip this approach when you need an archive of record, an access-controlled incident system, or a model-quality score. Free access can change, and a complete tool set can still do the wrong work. Coverage of calls is not correctness of outcome.

Teams that already export exactly once into a collector they trust can assert the same set equality in that API and drop the file. The predicate travels. The script is only the form that fits when the first store you control is a directory.

Run the gate once per run id, after the flush you already wait on. A non-zero exit blocks inspection. It does not indict the product. Store planned_count, exported_count, missing, and extra, then open a remote view only if that file says closed.

Refuse the Next Attempt While the Trace File Is Shapeless

Sam Sun — Sat, 10 Oct 2026 19:17:43 +0000

A second agent attempt is a cost, not a diagnosis, until the first trace file has a closed shape. The summary line can say the run finished, and that sentence still does not prove a root span exists, that each tool call has a child, or that an error carries a type you can group later. Treat the trace file as a packing list. If a label is missing, do not order another shipment.

Debug loops fail quietly when the artifacts do not share an id. The model returns a paragraph, a tool log shows a command, and the working tree may change, yet none of those three is a span. The next call then starts a new story, and you cannot tell whether it retried the same tool, skipped it, or invented a third path. A shape check sits in front of that call: it does not decide whether a patch is correct, only whether the run is comparable at all.

Shape is the packing list, not the cargo. A clerk can confirm a crate has a label, a weight, and a seal without opening every item. Presence of a trace id, unique span ids, parent links, a non-negative duration, a tool name, and an input hash is that seal. Whether the hash matches the file you meant to edit is a later question, and mixing the two is how a team spends a model call on a missing field.

Keep the contract small enough to fail for one reason: one file, one trace id, exactly one root whose parent is empty, and every other span pointing at a parent that also appears in the file. Span ids do not repeat, and end_ns is at least start_ns, which only checks that single span's own clock values rather than ordering across machines. Spans named with a tool. prefix carry tool.name and input.sha256, while a status of error carries an exception event or an error.type attribute, and the root carries run.id. A missing field is an exit code, not a warning scrolled past in CI.

Exit code 2 should stop the shell the way a failed test stops a deploy. A warning still lets the next step run, and the next step is the model call you were trying to protect. Exit code 0 means the file can be joined to a later attempt. It does not mean the agent fixed the bug, and it does not mean a remote collector has the same bytes.

The checker below is a proposed local tool: it has not been timed, and it is not a benchmark. It reads a simplified JSON export rather than OTLP protobuf. If your tracer already writes OTLP JSON, map spanId, parentSpanId, and startTimeUnixNano in a short adapter, then keep the gate itself dull. One failure reason per run is the point.

#!/usr/bin/env python3
"""Proposed trace-shape gate. Unexecuted example; adapt names to your exporter."""
import hashlib, json, sys

def fail(msg: str) -> None:
    print(f"shape: {msg}", file=sys.stderr)
    raise SystemExit(2)

def main() -> None:
    path = sys.argv[1] if len(sys.argv) > 1 else "run.json"
    with open(path, encoding="utf-8") as fh:
        doc = json.load(fh)
    spans = doc.get("spans")
    if not isinstance(spans, list) or not spans:
        fail("spans missing")
    ids = [s.get("span_id") for s in spans]
    if any(not i for i in ids) or len(ids) != len(set(ids)):
        fail("span_id missing or duplicated")
    roots = [s for s in spans if not s.get("parent_span_id")]
    if len(roots) != 1:
        fail(f"expected 1 root, found {len(roots)}")
    if not doc.get("trace_id"):
        fail("trace_id missing")
    known = set(ids)
    for s in spans:
        try:
            if int(s["end_ns"]) < int(s["start_ns"]):
                fail(f"{s['span_id']} negative duration")
        except (KeyError, TypeError, ValueError):
            fail(f"{s.get('span_id')} bad timestamps")
        parent = s.get("parent_span_id")
        if parent and parent not in known:
            fail(f"{s['span_id']} parent not in file")
        attrs = s.get("attributes") or {}
        if s is roots[0] and not attrs.get("run.id"):
            fail("root missing run.id")
        if str(s.get("name", "")).startswith("tool."):
            if not attrs.get("tool.name") or not attrs.get("input.sha256"):
                fail(f"{s['span_id']} tool span missing name or input hash")
        if s.get("status") == "error":
            events = [e.get("name") for e in (s.get("events") or [])]
            if "exception" not in events and not attrs.get("error.type"):
                fail(f"{s['span_id']} error has no type")
    digest = hashlib.sha256(",".join(ids).encode()).hexdigest()[:12]
    print(f"shape_ok trace={doc['trace_id']} spans={len(spans)} id_hash={digest}")

if __name__ == "__main__":
    main()

Commit a fixture next to the script so the gate can be tested without an agent. The root needs run.id, and the tool span needs a hex input.sha256, status error, and error.type set to something like timeout. python3 trace_shape.py fixture.json should print one shape_ok line and exit 0; strip input.sha256 into a copy and expect exit 2. If the broken copy still exits 0, the gate is decorative and should not sit in front of any model call.

python3 trace_shape.py fixture.json; echo "pass:$?"
python3 - <<'PY'
import json
d = json.load(open("fixture.json", encoding="utf-8"))
d["spans"][1]["attributes"].pop("input.sha256", None)
json.dump(d, open("/tmp/broken.json", "w"), indent=2)
PY
python3 trace_shape.py /tmp/broken.json; echo "broken:$?"

The fixture itself can stay under thirty lines. Two spans are enough to prove the root rule and the tool rule. Add a third span later only when you have a second tool. Extra spans that exist only to make the file look serious will train you to ignore the checker.

{
  "trace_id": "4f0c9a11c0de4b0a9c11aa00bb11cc22",
  "spans": [
    {
      "span_id": "aa",
      "parent_span_id": null,
      "name": "agent.run",
      "start_ns": 1000,
      "end_ns": 9000,
      "status": "ok",
      "attributes": {"run.id": "run-001"},
      "events": []
    },
    {
      "span_id": "bb",
      "parent_span_id": "aa",
      "name": "tool.read_file",
      "start_ns": 1500,
      "end_ns": 4000,
      "status": "error",
      "attributes": {
        "tool.name": "read_file",
        "input.sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
        "error.type": "timeout"
      },
      "events": [{"name": "exception"}]
    }
  ]
}

Store the raw tool argument outside the span, keyed by that hash, if you still need the body for a local replay. The gate only checks that the hash field is present and does not recompute it, because the body is not in this file. Recomputation belongs in a different script so a redaction change is not mistaken for an agent change. The id_hash printed on success is only a fingerprint of span ids in file order, useful in a CI log, and it will change if you sort spans differently.

A behavioral diff is a second step, and it should wait until two files pass, at which point you compare names, status codes, and input.sha256 values. A name that exists only in the second file is a new tool call, and a hash that changes under the same tool name is a new argument. Comparing two files that failed the gate produces a neat table of absences, which feels like analysis and is not. This gate stops before that comparison on purpose.

Skip the approach when a collector already enforces the same schema and the agent cannot emit a local file, because a weaker second gate will drift from the first. Skip it for sampled production traffic too: a sampler that drops tool children will make the checker fail runs you chose to thin out, and the exit code will blame instrumentation for a sampling decision. Skip it if exit code 0 would be shown to anyone as proof the working tree is correct, since a well-shaped trace can describe a wrong edit. Skip it if you cannot keep even a hash of tool arguments, because that hash is still a join key and a join key needs an owner and a retention rule.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access matters here only as the budget a shapeless retry would waste, and the free server option matters only as a later export target after trace_shape.py has already passed on a local file, so collector setup and agent behavior are not debugged in the same hour. No model name, quota, hardware size, or retention period is stated, because none of those were verified for this draft; if that server is unreachable, keep the JSON, since a remote batch that never arrives neither creates nor erases a local pass. Once the file is green, that free access is a reasonable place for the next attempt, as long as the checked file stays the source of record.

The loop stays short on purpose: emit one JSON file, run the shape script, and repair instrumentation until the broken fixture exits 2 and the real file exits 0. Then, and only then, spend another model call. The summary line can wait. The packing list cannot.

Hold the Score Until the Collector Acks the Link Set

Sam Sun — Fri, 09 Oct 2026 15:38:14 +0000

A split-host agent run is not ready to score until the collector acknowledges every linked span you expected to export. Process exit only proves the local loop stopped. The trace you will compare is whatever a remote collector actually stored.

That gap shows up as soon as the model hop and the tool hop do not share a process. The model request is created on a service you do not parent. The tool span is created beside the working tree. A third host, the collector, knows only what arrived.

Scoring from the local log grades a draft. This note treats the gap as a hard gate. The code below is a proposed workflow, not an executed benchmark, and it reports no measured miss rate.

A parent pointer is the wrong join

Parent and child spans assume one process tree. You do not have that tree when the model call leaves the machine. Writing a local parent id onto a remote request invents a hierarchy the collector cannot check.

The relation you can defend is a link. This tool span is associated with that remote request id, and both ids must be present before a score is written. A link does not pretend the remote process was your child. It records a join key you can test.

The picture is a warehouse manifest beside a dock receipt. The manifest lists what left. The receipt lists what the dock accepted. A diff that reads only the manifest will count crates still on the truck.

A successful model status does not close the receipt. It says the model answered. It does not say the tool span, the link, and the model span landed in one trace. Wall-clock order does not close it either, because the two hosts do not share a clock you control.

Fingerprints that survive a model swap

A low-cost model swap helps a debug loop, and it ruins a naive diff. If the span name embeds a model id, two runs that called the same tools look unrelated. The label moved. The tool contract may not have moved at all.

Key the tool contract, not the vendor string. Canonicalize the tool name and the arguments, then hash them. Put the hash on the local span, and hash the result only when it is small and non-sensitive.

The comparison then asks whether the fingerprint sequence changed. It does not ask whether the model label changed. Stable hashes keep a model swap from looking like a regression.

Canonical form is part of the method. Sort keys, reject non-finite floats, and freeze separators. Otherwise a harmless reorder looks like a behavior change. Keep the canonicalizer in the same file as the gate so the expected set and the later diff cannot drift.

Raw arguments should not ride to a shared collector when they might hold secrets. The fingerprint can travel. The body stays in a local file you open only after a hash differs. That split is a limit of this loop, not a full redaction program.

The gate, as proposed code

The script has not been run for this article. It encodes one rule. Do not emit a ready score unless the ack covers every expected id, including the remote request id. Save it as ack_gate.py if you want the shell step below to import it. The type hints assume Python 3.9 or newer.

#!/usr/bin/env python3
"""Proposed gate. Unexecuted example: hold the score until the ack covers the set."""

import hashlib
import json
import sys
from pathlib import Path

def canonical(payload: dict) -> bytes:
    return json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
        allow_nan=False,
    ).encode()

def fingerprint(payload: dict) -> str:
    return hashlib.sha256(canonical(payload)).hexdigest()

def build_expected(run_id: str, remote_request_id: str, calls: list) -> dict:
    spans = []
    for index, call in enumerate(calls):
        spans.append({
            "span_id": f"{run_id}-tool-{index:03d}",
            "run_id": run_id,
            "link": {"remote_request_id": remote_request_id},
            "tool_fingerprint": fingerprint({"name": call["name"], "args": call["args"]}),
            "result_fingerprint": fingerprint({"result": call.get("result")}),
        })
    return {
        "run_id": run_id,
        "remote_request_id": remote_request_id,
        "spans": spans,
    }

def score_allowed(expected: dict, ack: dict) -> bool:
    wanted = {span["span_id"] for span in expected["spans"]}
    wanted.add(expected["remote_request_id"])
    return wanted <= set(ack.get("accepted_ids", []))

def main() -> int:
    expected = json.loads(Path(sys.argv[1]).read_text())
    ack = json.loads(Path(sys.argv[2]).read_text())
    if not score_allowed(expected, ack):
        wanted = {span["span_id"] for span in expected["spans"]}
        wanted.add(expected["remote_request_id"])
        missing = sorted(wanted - set(ack.get("accepted_ids", [])))
        print(json.dumps({"status": "hold", "missing": missing}))
        return 2
    sequence = [span["tool_fingerprint"] for span in expected["spans"]]
    print(json.dumps({
        "status": "ready",
        "run_id": expected["run_id"],
        "sequence": sequence,
    }))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Build the expected file from the calls you actually made, before you export. The numbers in the fixture are placeholders, not measurements.

python3 - <<'PY'
import json
from ack_gate import build_expected

calls = [
    {"name": "read_file", "args": {"path": "src/app.py"}, "result": {"bytes": 1204}},
    {"name": "run_tests", "args": {"target": "tests/test_app.py"}, "result": {"failed": 0}},
]
doc = build_expected("run-1842", "req-9f3a", calls)
open("expected.json", "w").write(json.dumps(doc, indent=2) + "\n")
PY
python3 ack_gate.py expected.json ack.json

Exit 2 means hold, and the JSON lists missing ids. Exit 0 means the fingerprint sequence may be compared with another ready run. A 202 from an async ingest URL is a queue ticket, not an ack file.

Persist the ack only from a body that names accepted ids. Include the remote request id in the same expected set as the tool spans. A tool-only ack still hides the model side, so the gate holds.

Span events are not a substitute for that body. An event can be sampled away while the span id still arrives, or the reverse. The score waits on ids the collector lists, not on events you hoped to attach.

Walk one failed export before you trust the script. Suppose the model span arrives and the second tool span does not, because the batch flushed early. The missing list contains run-1842-tool-001, and the sequence is withheld.

Retry the export with the same span ids. Replace ack.json with the new body, then run the gate again. Do not average the two acks, and do not mark the run partial-ready. Partial-ready is how a later diff blames the model for a span that never landed.

Idempotency belongs on the export, not on a fresh id per attempt. Reuse run-1842-tool-001 when you resend. A new id on each retry makes the expected set chase a moving target, and the ack can never catch it.

Where the free path fits

You can practice the split without standing up a private collector. MonkeyCode is the open-source project used here as the remote side: free model access for the model hop, and a free server option for the collector hop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both are availability claims, not a capacity plan. This article does not name models, token quotas, machine sizes, retention, or uptime. Those figures move, and a stale quota in a debug script is worse than no quota at all.

Read the current project documentation before you bind the loop to a limit. Do not paste an old token budget into the gate. If the docs and the runtime disagree, trust the runtime error and update the note.

Point the model hop at the free model access. Point the exporter at the free server, and send ids plus fingerprints rather than raw tool bodies. Keep expected.json beside the run directory so review does not depend on a dashboard session.

If the server schema cannot list accepted ids, do not fill them in locally. Extend the export until the ack is explicit, or leave the run on hold. A locally invented ack is just the manifest copied onto receipt paper.

The review step stays on the machine that ran the tools. When a ready sequence changes, open the local tool log for that run id and note the first hash that moved. The shared server is an index. It does not declare the agent fixed.

Who should not use this

The gate catches a missing export. It does not catch a wrong result that was exported in full. You still need an oracle on the fingerprint or on the local outcome.

It assumes the ack is specific and repeatable. If the server only says ok, the script holds forever, unless someone stubs accepted_ids with the expected set and voids the check. Do not stub it.

Do not use a free server option as audit storage, incident evidence, or a system of record. A convenience collector can drop or reset data. Do not send tool arguments that contain credentials, customer data, or private source.

Skip the gate if the model and the tools already share one process and one tracer. The extra files are ceremony you will ignore. Skip it if you cannot change the exporter to name accepted ids.

A gate without an honest ack is a new way to feel finished. Waiting costs whatever the batch interval costs. That wait is the method. A faster score on a partial trace is a confident miss.

Close on three files

Keep the expected set, the collector ack, and the fingerprint sequence. Diff sequences only across runs whose acks covered their sets. If the sequence is stable and the local oracle passed, record the run as unchanged.

If the sequence moved, inspect the local body for the first differing hash and stop. Do not keep walking the trace until you find a story that fits. The first hash delta is the debug pointer.

That is smaller than a claim that the agent is fixed. It is a claim you can recheck after the model label changes and the tool contract does not. A later swap should move the vendor string and leave the sequence alone, or the contract really changed.

If run notes already sit in the working tree, write the expected set there and export only ids and fingerprints. Recheck the current free model access and free server terms in the project docs. Let the ack file, not a status page, decide whether the run is scorable.