DEV Community: Sam Sun

Unfingerprinted Generation Spans Cannot Be Diffed

Sam Sun — Sun, 20 Sep 2026 12:02:51 +0000

A trace diff is undefined until every generation span carries a fingerprint of the prompt, the tool schema, and the declared model identity. Output tests do not supply that. Neither does a green exit code.

When an agent run looks worse overnight, the usual reflex is to rewrite the system prompt. That reflex assumes the two runs were the same experiment. They often were not. Shared free-model endpoints omit model, rewrite routing, or inject a provider preamble you never hashed. The second run is a different instrument. Comparing it to the first is like subtracting voltages measured on two unlabeled probes.

The measurement problem is older than agents. A unit test that asserts on the final file will pass while the model retried a tool four times, dropped a call_id, or answered from a different checkpoint than yesterday. Process and product diverged. If you only keep the product, you cannot say which one moved.

A generation fingerprint is a small, stable tuple stored as span attributes. It is not a dashboard slogan. Four fields are enough to make two traces comparable, or to prove they are not.

prompt.hash is a truncated SHA-256 of canonical JSON: system text, user text, and the tool JSON schema. tool.schema_hash is the same hash over the tools array alone, so a prompt-only edit does not look like a tool-surface change. gen_ai.request.model is the declared identifier if the exporter emitted one. If it is missing, the span is incomparable by default. gen_ai.request.temperature belongs in the tuple because a silent 0.0 to 0.7 swap will scatter tool-call order without touching the prompt.

Hash canonicalization must be boring and strict. Sort keys. Reject NaN. Encode UTF-8. Do not pretty-print. Two semantically equal prompts that differ by a trailing newline are different experiments, and the hash should say so.

The following module is a template. Wire it to your own span exporter. It has not been executed against a public corpus in this article.

# fingerprint.py — template, not a vendor SDK
from __future__ import annotations

import hashlib
import json
from typing import Any

def canonical(obj: Any) -> bytes:
    return json.dumps(
        obj,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
        allow_nan=False,
    ).encode("utf-8")

def short_sha(obj: Any, n: int = 16) -> str:
    return hashlib.sha256(canonical(obj)).hexdigest()[:n]

def generation_fingerprint(
    *,
    system: str,
    user: str,
    tools: list[dict],
    model: str | None,
    temperature: float | None,
) -> dict[str, str]:
    prompt_payload = {"system": system, "user": user, "tools": tools}
    attrs = {
        "prompt.hash": short_sha(prompt_payload),
        "tool.schema_hash": short_sha(tools),
        "fingerprint.missing_model": "true" if not model else "false",
    }
    if model:
        attrs["gen_ai.request.model"] = model
    if temperature is not None:
        attrs["gen_ai.request.temperature"] = str(temperature)
    attrs["fingerprint.tuple"] = short_sha(attrs)
    return attrs

Attach those attributes on the generation span at start, not after the model returns. A fingerprint written on the way out can include the completion and will then change when the model is flaky. You want the request identity, not the answer identity. Completion hashes belong on a child event if you need them.

Free shared endpoints make the missing-model case common. The HTTP layer answers. The span does not name a checkpoint. Re-running the same prompt on another host will not repair that. Refuse comparability until the exporter records an identifier, or pin one yourself from configuration rather than from the response body. Reading the model name back from the completion is how silent swaps leak into “stable” traces.

# stamp_model_from_config.py — template
import json
import os
import sys

declared = os.environ.get("AGENT_MODEL_ID", "").strip()
if not declared:
    print("AGENT_MODEL_ID unset; refusing export", file=sys.stderr)
    sys.exit(2)

for line in sys.stdin:
    span = json.loads(line)
    if span.get("kind") == "generation" and not span.get("gen_ai.request.model"):
        span["gen_ai.request.model"] = declared
        span["fingerprint.missing_model"] = "false"
        span["fingerprint.model_source"] = "config"
    sys.stdout.write(json.dumps(span, ensure_ascii=False) + "\n")

export AGENT_MODEL_ID=from-config-not-from-response
python stamp_model_from_config.py < raw.ndjson > stamped.ndjson

fingerprint.model_source=config is an admission, not a boast. It records that the name did not come from the provider. Downstream diffs can still run. They must not pretend the checkpoint was observed.

Export two runs as JSON-lines, one span per line. The comparer below does not grade quality. It answers a narrower question: may these traces be diffed at all?

# compare_fingerprints.py — template
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = ("prompt.hash", "tool.schema_hash")

def load_generations(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        span = json.loads(line)
        if span.get("kind") != "generation":
            continue
        rows.append(span)
    return rows

def comparable(a: dict, b: dict) -> tuple[bool, str]:
    if a.get("fingerprint.missing_model") == "true" or b.get("fingerprint.missing_model") == "true":
        return False, "missing_model"
    keys = REQUIRED + ("gen_ai.request.model", "gen_ai.request.temperature")
    for key in keys:
        if a.get(key) != b.get(key):
            return False, f"mismatch:{key}"
    return True, "ok"

def main(path_a: str, path_b: str) -> int:
    gens_a = load_generations(Path(path_a))
    gens_b = load_generations(Path(path_b))
    if not gens_a or not gens_b:
        print("no generation spans", file=sys.stderr)
        return 3
    # Index alignment is a first pass. Prefer generation_id in production.
    n = min(len(gens_a), len(gens_b))
    incomparable = 0
    for i in range(n):
        ok, why = comparable(gens_a[i], gens_b[i])
        if not ok:
            incomparable += 1
            print(f"span[{i}] incomparable ({why})")
    if len(gens_a) != len(gens_b):
        print(f"generation count {len(gens_a)} != {len(gens_b)}")
        return 2
    if incomparable:
        return 2
    print("fingerprints align; a path diff is defined")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run it as a gate in front of any path diff you already keep.

python compare_fingerprints.py run-a.ndjson run-b.ndjson
echo $?   # 0 defined, 2 incomparable, 3 missing generations

Exit code 2 is not a product failure. It is a lab-setup failure. Do not open a prompt PR. Do not bump a temperature. Restore the fingerprint first.

The decision rule is small enough to keep in the comparer, not in a wiki. If fingerprint.missing_model is true on either side, the traces are incomparable. If prompt.hash differs, you changed the experiment; attribute the delta to the prompt, not to “the agent.” If only tool.schema_hash differs, the model saw a different action surface. If only gen_ai.request.model differs, you are looking at a routing change. If the tuple matches and the child tool graph does not, then—and only then—you have an agent-path delta worth debugging.

That last case is the only one in which a span-level path diff has meaning. Everything else is a contaminated A/B test. Teams that skip the gate will spend the week “fixing” a prompt that never moved.

If you need a second machine to collect the B run, a free server option is enough to host the exporter and the comparer. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Treat both as a re-run host for the fingerprint gate above, not as a substitute for the attributes. After the spans exist, the method does not depend on a particular product.

Limitations are sharp. A hash does not see a provider-side system preamble that never entered your canonical JSON. Two fingerprints can match while safety filters rewrite the completion. Temperature in the tuple does not capture nucleus sampling or frequency penalty if you never record them. Index alignment of generation spans is a weak join. Production code should join on a client-generated generation_id. The templates above are a specification for exit codes, not a benchmark.

Do not store raw prompts next to the hashes if those prompts contain secrets. The hash is for equality, not for recovery. Do not use the fingerprint as a billing key. Do not sample generation spans before hashing. A 1% sample will drop the only span whose model identity was missing and will fake comparability.

Skip this workflow if every run already pins a model digest, a signed prompt bundle, and a tool-schema version in CI. You already have a stronger fingerprint, and SHA-16 on the side is noise. If you have no traces yet, record closed generation spans first. Hashing empty files will only certify that two voids are equal. If the agent is a single non-tool completion graded by a human, a prompt hash plus an output rubric is enough and the comparer is overhead.

The reusable debug loop is then four beats, always in this order. Fingerprint. Refuse the diff when the tuple disagrees. Diff tool children only after a defined comparison. Only then touch the prompt. Reverse the order and you will keep shipping prompt edits that chase routing noise.

If you already export spans, add the four attributes before you add another chart.

Join Tool Exits to Spans Before Trusting the Summary

Sam Sun — Sat, 19 Sep 2026 10:08:21 +0000

A model’s closing paragraph is not a trace. The only agent run you can debug is the one where every tool process exit inner-joins to a closed span on a caller-issued invocation_id, and where span status is a function of that exit code rather than of later assistant text.

That rule sounds pedantic until a coding agent shells out twice, retries once, and then writes “ran the tests.” The summary is one sentence. The machine did three things. Two of them may have failed. If those failures never became spans, the debug loop starts from fiction.

Think of baggage tags. A passenger name on a boarding pass does not locate a suitcase. The tag number does. Tool names in a chat transcript are passenger names. invocation_id is the tag. Order of appearance in the log is not a join key. Concurrent tools scramble wall-clock order, and retries mint extra exits that never show up in the last message.

What the join actually asserts

Treat the run as two relations that must agree. Relation E is process exits: one row per OS-level tool invocation, with invocation_id, exit_code, ended_at, and a hash of stdout plus stderr. Relation S is closed spans: one row per span that claimed to wrap a tool, with the same invocation_id, a mapped span_status, and closed_at.

A run is join-complete when three predicates hold. Every row in E matches exactly one row in S. Every row in S matches exactly one row in E. For each matched pair, span_status equals the mapping of exit_code (zero to ok, anything else to error). Fail the run on leak, orphan, or mismatch. Do not “warn and continue.” A warning is how unmatched exits become folklore.

The mapping is deliberately crude. A test harness that prints FAILED and still exits 0 is a product bug in the tool, not a reason to let the model overwrite span status. If you need richer outcomes, put them on span events. Status stays a function of the process.

A proposed local harness

The checker below is a proposed local harness, not a production trace backend. It writes JSONL. It does not sample. Label it as unexecuted in your environment until you run it against a real agent wrapper.

# join_exits.py — proposed checker, file-backed JSONL
from __future__ import annotations

import hashlib, json, os, subprocess, time, uuid
from pathlib import Path

TRACE = Path(os.environ.get("TRACE_JSONL", "run.jsonl"))

def _write(row: dict) -> None:
    row["ts"] = time.time()
    with TRACE.open("a", encoding="utf-8") as f:
        f.write(json.dumps(row, sort_keys=True) + "\n")

def map_status(exit_code: int) -> str:
    return "ok" if exit_code == 0 else "error"

def run_tool(argv: list[str]) -> int:
    invocation_id = str(uuid.uuid4())
    env = os.environ.copy()
    env["INVOCATION_ID"] = invocation_id
    _write({"type": "span_open", "invocation_id": invocation_id, "argv": argv})
    proc = subprocess.run(argv, env=env, capture_output=True)
    payload = proc.stdout + b"\0" + proc.stderr
    digest = hashlib.sha256(payload).hexdigest()
    status = map_status(proc.returncode)
    _write({
        "type": "process_exit",
        "invocation_id": invocation_id,
        "exit_code": proc.returncode,
        "stdout_stderr_sha256": digest,
        "bytes": len(payload),
    })
    _write({
        "type": "span_close",
        "invocation_id": invocation_id,
        "span_status": status,
        "exit_code": proc.returncode,
    })
    return proc.returncode

def load_rows() -> list[dict]:
    if not TRACE.exists():
        return []
    return [json.loads(line) for line in TRACE.read_text().splitlines() if line.strip()]

def join_report(rows: list[dict]) -> dict:
    exits = {r["invocation_id"]: r for r in rows if r.get("type") == "process_exit"}
    closes = {r["invocation_id"]: r for r in rows if r.get("type") == "span_close"}
    leaks = sorted(set(exits) - set(closes))
    orphans = sorted(set(closes) - set(exits))
    mismatches = sorted(
        i for i in set(exits) & set(closes)
        if closes[i]["span_status"] != map_status(exits[i]["exit_code"])
    )
    return {
        "exit_count": len(exits),
        "close_count": len(closes),
        "leaks": leaks,
        "orphans": orphans,
        "status_mismatches": mismatches,
        "join_ok": not (leaks or orphans or mismatches),
    }

if __name__ == "__main__":
    import sys
    cmd = sys.argv[1:]
    if cmd[:1] == ["--report"]:
        report = join_report(load_rows())
        print(json.dumps(report, indent=2))
        raise SystemExit(0 if report["join_ok"] else 2)
    if not cmd:
        raise SystemExit("usage: join_exits.py <tool> [args...] | --report")
    raise SystemExit(run_tool(cmd))

Wrap the agent’s tool runner so it cannot exec except through run_tool. A one-line shell stand-in is enough to prove the join on a laptop:

export TRACE_JSONL="$PWD/run.jsonl"
: > "$TRACE_JSONL"
python join_exits.py python -c "print('ok')"
python join_exits.py python -c "raise SystemExit(1)"
python join_exits.py --report

The second command must produce span_status of error. If your agent later prints “all good,” the report still fails. That is the point. The summary is not a source row in either relation.

How summaries lie without leaking bytes

Three failure modes show up constantly, and none of them require a sophisticated model. The agent retries a formatter, the first process exits 1, the second exits 0, and only the second is wrapped in a span. Relation E has two rows. Relation S has one. The leak is the first exit. The summary mentions a single format step because language models compress.

The inverse also happens. A span is opened for pytest, the process is never started, and the span is closed ok because the orchestrator timed out of the thought loop and needed a status. That is an orphan. It is worse than a leak. A leak hides work. An orphan invents work.

Status mismatch is quieter. The process exits 2. A post-processor, or the model itself, writes span_status: ok because tests were “mostly fine.” The inner join on invocation_id succeeds. The run still has to fail. Id equality is not semantic equality.

If you want a regression hook, do not grep the assistant message. Assert the report:

python join_exits.py --report
test "$(python -c 'import json,sys; print(json.load(sys.stdin)["join_ok"])' < <(python join_exits.py --report))" = True

Better: read the JSON. Gate CI on join_ok, exit_count, and a ceiling on error closes. A bound on tool exits is an observability SLO. It is not a token budget. Token counters live in another pipeline and do not prove that a process was traced.

Where a free model host fits, and where it does not

The join does not need a particular vendor. It needs a wrapper you control. When the loop also needs a model to propose the next tool, standing up paid inference just to debug tracing is a poor trade. MonkeyCode is an open-source project that currently offers free model access and a free server option for that kind of harness work. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep secrets off that path. The checker hashes stdout and stderr; it does not redact them at the source. If a tool prints a token or a customer fixture, the JSONL becomes a credential store. Run the free server option only on synthetic repos and fixtures you would paste into a ticket. The join logic itself should stay local, next to the working tree.

Strip the product out and the method is unchanged. You still emit span_open, process_exit, and span_close. You still fail on leaks. That is the usefulness test.

Limitations, and who should skip this

This join assumes tools are processes. In-process function calls that never exec will not appear in relation E unless you fabricate an exit row. Fabricated exits are how orphans are born. If your agent’s tools are Python callables, instrument the callable, not a fake subprocess.

It also assumes one exit per invocation_id. Pipelines that reuse an id across retries will collapse two exits into one dict key in the sample checker. Retries need new ids. Correlate them with a parent_invocation_id field if you must, but do not recycle the primary key.

Do not use this as a substitute for OpenTelemetry if you already have a collector, span processors, and identity at the RPC layer. Dual-writing JSONL and OTLP without a single id space creates a second forest. Do not use it on jobs whose stdout is legally sensitive. Hashing does not erase the bytes you wrote before the hash.

Clock skew is out of scope. The join key is the id, not ended_at ≈ closed_at. If you add a time window as a hint, keep it diagnostic. Never let a 50ms skew decide that an exit “probably” belongs to a span with a different id.

The last assistant message can still be useful after the join passes. Read it as commentary. Commentary is not a row. If the join fails, do not debate the commentary. Fix the wrapper until leaks, orphans, and status_mismatches are empty lists, then read the prose.

If your traces are already JSONL, run join_exits.py --report on the last failed job before you add another log line. Extra logs without a join key are more passenger names. They will not find the bag.

Wall-Clock Minus Exclusive Spans Is the Untraced Remainder

Sam Sun — Fri, 18 Sep 2026 08:35:22 +0000

An agent run that lasts eight seconds is not an eight-second tool. The number that should gate a retry, a budget, or a claim that the loop did engineering work is the untraced remainder: wall-clock duration minus the union of closed spans, after children have been subtracted from parents. Everything else is a mood.

Most debug loops still print elapsed time and move on. That figure mixes model wait, overlapping tools, process startup, and gaps where the tracer was not listening. Treat it as cost and you will “optimize” a parallel read that was already cheap, or miss a two-second hole where the model sat idle with no span at all.

Inclusive duration is a nested Russian doll. A parent span that wraps a plan step and three tools will report the whole doll. Sum those parents with their children and you double-count the inner work. Parallel tools make it worse: two 400 ms reads that overlap are not 800 ms of exclusive machine time. They are closer to 400 ms of wall and 800 ms of billed tool occupancy, and those are different questions.

A restaurant bill is a fair analogy. Summing dish prices ignores tax, shared plates, and the twenty minutes you waited for a table. Wall-clock is the evening. Inclusive span sum is every dish counted with the table time glued on. Exclusive span math is the only way to see what each course actually occupied.

The contract is small. Every span must have a start, an end, a parent id or a root, and a kind that is either model or tool. If end_ms is missing, the run is not closed. If two siblings overlap, their inclusive times must not be added. If the union of all intervals is much smaller than the process lifetime, the remainder is untraced work, not a rounding error.

The artifact below is a single-file judge. It is a proposal you can run locally; it is not a production tracer. Feed it a JSON run, and it prints exclusive durations, overlap pairs, and the untraced remainder as a fraction of wall-clock.

#!/usr/bin/env python3
"""exclusive_span.py — attribute agent time after exclusive span math."""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class Span:
    span_id: str
    parent_id: str | None
    kind: str
    name: str
    start_ms: int
    end_ms: int

    @property
    def inclusive_ms(self) -> int:
        return self.end_ms - self.start_ms


def load_run(path: str) -> tuple[int, int, list[Span]]:
    with open(path, encoding="utf-8") as fh:
        raw = json.load(fh)
    spans = []
    for row in raw["spans"]:
        if row.get("end_ms") is None:
            raise ValueError(f"unclosed span {row.get('span_id')}")
        if row["end_ms"] < row["start_ms"]:
            raise ValueError(f"inverted span {row['span_id']}")
        spans.append(
            Span(
                span_id=row["span_id"],
                parent_id=row.get("parent_id"),
                kind=row["kind"],
                name=row["name"],
                start_ms=int(row["start_ms"]),
                end_ms=int(row["end_ms"]),
            )
        )
    return int(raw["started_at_ms"]), int(raw["ended_at_ms"]), spans


def children_of(spans: list[Span], parent_id: str) -> list[Span]:
    return [s for s in spans if s.parent_id == parent_id]


def exclusive_ms(span: Span, spans: list[Span]) -> int:
    kids = children_of(spans, span.span_id)
    covered = union_ms((k.start_ms, k.end_ms) for k in kids)
    clipped = 0
    for a, b in covered:
        lo = max(a, span.start_ms)
        hi = min(b, span.end_ms)
        if hi > lo:
            clipped += hi - lo
    return span.inclusive_ms - clipped


def union_ms(intervals: Iterable[tuple[int, int]]) -> list[tuple[int, int]]:
    ordered = sorted(intervals)
    if not ordered:
        return []
    out = [ordered[0]]
    for start, end in ordered[1:]:
        last_s, last_e = out[-1]
        if start <= last_e:
            out[-1] = (last_s, max(last_e, end))
        else:
            out.append((start, end))
    return out


def overlap_pairs(spans: list[Span]) -> list[tuple[str, str, int]]:
    pairs = []
    for i, a in enumerate(spans):
        for b in spans[i + 1 :]:
            if a.parent_id != b.parent_id:
                continue
            lo = max(a.start_ms, b.start_ms)
            hi = min(a.end_ms, b.end_ms)
            if hi > lo:
                pairs.append((a.span_id, b.span_id, hi - lo))
    return pairs


def remainder_ms(start: int, end: int, spans: list[Span]) -> int:
    covered = union_ms((s.start_ms, s.end_ms) for s in spans)
    covered_total = sum(b - a for a, b in covered)
    return max(0, (end - start) - covered_total)


def report(path: str) -> None:
    start, end, spans = load_run(path)
    wall = end - start
    rem = remainder_ms(start, end, spans)
    print(f"wall_ms={wall}")
    print(f"untraced_ms={rem} fraction={rem / wall if wall else 0:.3f}")
    for span in spans:
        print(
            f"{span.span_id} kind={span.kind} name={span.name} "
            f"inclusive={span.inclusive_ms} exclusive={exclusive_ms(span, spans)}"
        )
    for a, b, ms in overlap_pairs(spans):
        print(f"overlap {a} {b} ms={ms}")
    model_ex = sum(exclusive_ms(s, spans) for s in spans if s.kind == "model")
    tool_ex = sum(exclusive_ms(s, spans) for s in spans if s.kind == "tool")
    print(f"exclusive_model_ms={model_ex} exclusive_tool_ms={tool_ex}")
    if rem / wall > 0.25 if wall else False:
        print("FAIL: untraced remainder exceeds 25% of wall-clock")
        sys.exit(2)
    if tool_ex == 0 and wall > 0:
        print("FAIL: no exclusive tool time; this run is a chat, not a tool loop")
        sys.exit(3)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.stderr.write("usage: exclusive_span.py run.json\n")
        sys.exit(1)
    report(sys.argv[1])

A fixture makes the failure mode obvious. The process lives 8400 ms. Tools overlap. A 1800 ms gap has no span. Inclusive sums look busy. Exclusive math does not.

{
  "run_id": "demo-8400",
  "started_at_ms": 0,
  "ended_at_ms": 8400,
  "spans": [
    {"span_id": "m1", "parent_id": null, "kind": "model", "name": "plan",
     "start_ms": 50, "end_ms": 2100},
    {"span_id": "t1", "parent_id": "m1", "kind": "tool", "name": "read_a",
     "start_ms": 2200, "end_ms": 2650},
    {"span_id": "t2", "parent_id": "m1", "kind": "tool", "name": "read_b",
     "start_ms": 2300, "end_ms": 2700},
    {"span_id": "m2", "parent_id": null, "kind": "model", "name": "revise",
     "start_ms": 4500, "end_ms": 6200},
    {"span_id": "t3", "parent_id": "m2", "kind": "tool", "name": "patch",
     "start_ms": 6300, "end_ms": 6550}
  ]
}

Save the judge as exclusive_span.py and the fixture as run.json, then run it as a gate, not as a dashboard widget.

python3 exclusive_span.py run.json; echo exit:$?

On this fixture the wall is 8400 ms. The union of spans leaves a hole from 2700 to 4500, plus short edges at start and finish. Sibling tools t1 and t2 overlap by 350 ms, so adding their inclusive times invents 350 ms of serial work that never happened. Exclusive model time and exclusive tool time become the two columns you can actually budget. The script exits 2 because the untraced remainder is larger than a quarter of the run. That is the point.

A second command turns the same file into a regression check. Keep a known-good remainder ceiling next to the fixture. If a prompt change stretches the hole, the gate fails before anyone calls the agent “faster” because wall-clock happened to drop on a warm cache.

python3 - <<'PY'
import json, pathlib, subprocess, sys
raw = json.loads(pathlib.Path("run.json").read_text())
assert raw["ended_at_ms"] - raw["started_at_ms"] == 8400
proc = subprocess.run([sys.executable, "exclusive_span.py", "run.json"])
print("gate_status", proc.returncode)
PY

Use wall-clock when you care about user-visible wait. Use inclusive parent duration when you are debugging a single subtree and you want the doll, not the parts. Use exclusive duration when you allocate cost to a kind. Use the untraced remainder when you decide whether the tracer is complete enough to argue from. Mixing those four numbers is how teams convince themselves a chatty model is a fast toolchain.

The 25% remainder threshold in the script is a local policy, not a law. Tighten it on short, fully instrumented loops. Loosen it if your runtime injects GC pauses you do not span on purpose. Do not copy the constant into a vendor SLA. The useful part is the formula, not the cutoff.

Clock skew will lie. If tools run on another host and stamps are not on the same clock, overlap pairs become fiction and exclusive math subtracts the wrong children. Missing end_ms is worse: an open span makes inclusive duration undefined, so the script refuses the run rather than inventing a close. Async tools that finish after the parent has already closed will look like orphans; exclusive parent time will be too large and the remainder too small. A forest that is actually a DAG, with two parents claiming one child, is outside this model. Fix the shape first.

This debug loop is the wrong tool if you only have stdout lines, token counts, or a UI timer. It is also the wrong tool if your agent is a single model call with no tools; the second failure path will keep firing, and it should. Teams that need distributed tracing with baggage, sampling, and tail-based retention want an OpenTelemetry pipeline, not a hundred-line judge. The script exists to make a local run honest before you scale the lie.

Current public debate likes to separate “vibe” from engineering by tone. Duration attribution is blunter. A run whose exclusive tool time is zero did not touch the working tree in any spanned way. A run whose remainder is half the wall-clock did work you cannot replay. Neither case is fixed by a greener unit test that never saw the hole.

If you already emit closed spans from your own harness, the judge is enough. If you need a place to exercise the agent while you iterate on that harness, MonkeyCode’s free model access and free server option can host the loop so the remainder math stays local. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Ship the gate next to the fixture. When exclusive tool time rises and the untraced fraction falls, you have a trace you can budget. Until then, wall-clock is only how long you waited to be unsure.

Map Every Working-Tree Delta to a Closed Tool Span

Sam Sun — Thu, 17 Sep 2026 06:47:10 +0000

An agent run is finished when the working tree and the trace agree, not when the model emits a final sentence. git status --porcelain is the ledger. Closed tool spans are the journal entries. If a path changed and no span owns it, the run failed. If a span claims a write and the bytes are missing, the run failed. The summary paragraph is not evidence.

This gap is getting easier to miss. Public debate this month keeps returning to whether generated code is already “better than most developers,” and whether a green test suite is enough to call the session engineering. Those arguments skip a more primitive check. Tests observe outputs someone chose to assert. They do not observe every file the process touched. An agent can skip a tool, narrate the skip as success, and still pass a unit file that never opened the mutated path.

Think of a warehouse cycle count. The packing list the intern typed is the assistant message. The scanner at the dock is the trace. The pallets on the floor are git status. Nobody ships because the intern wrote “all packed.” You ship when every pallet id is on a scan record and every scan record points at a pallet that still exists. Agent tooling that treats the chat transcript as the dock scan will leak mutations into the next human commit.

The failure has two directions, and they are not symmetric. A ghost write is a dirty path with no closed span: a formatter, a shell redirect, a compiler, or a helper script that ran outside the traced adapter. A phantom write is a closed span whose after_hash is not on disk: the tool returned ok, the model thanked it, and a later step reverted the file, or the tool wrote a temp path and reported a source path. Either break should fail the run. A third, quieter break is a clean tree after a task that required a mutation. No-op success is still a false success if the prompt demanded an edit.

A reusable control is a bijection between path-level deltas and closed write spans. The harness below is a worked example, not a production trace backend. It records JSONL, hashes file bytes with SHA-256, and compares that set to git status --porcelain. It does not replace OpenTelemetry. It does claim that a run without this map is an anecdote.

Label the schema as an example. Each tool attempt appends one record. status must become closed before reconcile. op is write, delete, or read. Reads do not have to match porcelain. Writes and deletes do. path is repo-relative POSIX. before_hash and after_hash are hex or null.

# span_schema.py — example record shape, not a shipped SDK
import hashlib, json, time, uuid
from pathlib import Path

def sha256_file(path: Path):
    if not path.is_file():
        return None
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def open_span(trace_path, op, relpath, before_hash):
    rec = {
        "span_id": str(uuid.uuid4()),
        "ts_open": time.time_ns(),
        "op": op,
        "path": relpath,
        "before_hash": before_hash,
        "after_hash": None,
        "status": "open",
        "error": None,
    }
    with open(trace_path, "a", encoding="utf-8") as f:
        f.write(json.dumps(rec) + "\n")
    return rec["span_id"]

def close_span(trace_path, span_id, after_hash, error=None):
    rows = []
    with open(trace_path, encoding="utf-8") as f:
        for line in f:
            rec = json.loads(line)
            if rec["span_id"] == span_id:
                rec["ts_close"] = time.time_ns()
                rec["after_hash"] = after_hash
                rec["status"] = "error" if error else "closed"
                rec["error"] = error
            rows.append(rec)
    with open(trace_path, "w", encoding="utf-8") as f:
        for rec in rows:
            f.write(json.dumps(rec) + "\n")

Wrap every mutating tool. Do not wrap the model. The model can say anything. The adapter is the only process allowed to touch the tree. If the agent is given a raw shell, the shell itself must emit spans for every path it intends to change, or the later git diff will fail the run on purpose. That is the point. Untraced sed -i is not a clever escape. It is an open drawer.

# traced_write.py — example adapter
from pathlib import Path
from span_schema import sha256_file, open_span, close_span

def traced_write(repo: Path, trace_path: str, relpath: str, data: bytes):
    dest = (repo / relpath).resolve()
    if not str(dest).startswith(str(repo.resolve())):
        raise ValueError("path escapes repo")
    dest.parent.mkdir(parents=True, exist_ok=True)
    before = sha256_file(dest)
    span_id = open_span(trace_path, "write", relpath, before)
    try:
        dest.write_bytes(data)
        close_span(trace_path, span_id, sha256_file(dest))
    except Exception as exc:
        close_span(trace_path, span_id, sha256_file(dest), error=str(exc))
        raise

Collect the ledger from git, not from in-memory guesses. Porcelain is stable enough for a gate. Untracked files are mutations. Ignored files are a policy choice: if the agent is allowed to write node_modules, the bijection is already blind. Keep generated noise out of the repo, or accept that the map cannot see it.

# git_delta.py — example ledger
import subprocess
from pathlib import Path

def porcelain_paths(repo: Path) -> dict:
    out = subprocess.check_output(
        ["git", "status", "--porcelain", "-uall"],
        cwd=repo, text=True,
    )
    deltas = {}
    for line in out.splitlines():
        if not line:
            continue
        code, rest = line[:2], line[3:]
        if " -> " in rest:
            rest = rest.split(" -> ", 1)[1]
        deltas[rest.strip()] = code.strip()
    return deltas

Reconcile is a set comparison plus a hash check. Open spans fail first. Then every write or delete span must match a porcelain path. Then every porcelain path must match a span. Then after_hash must equal the bytes on disk for writes, and the path must be absent for deletes. Print the leftover sets. Do not summarize them in prose and call the run “mostly fine.”

# reconcile.py — example gate
import json, sys
from pathlib import Path
from span_schema import sha256_file
from git_delta import porcelain_paths

WRITE_OPS = {"write", "delete"}

def load_spans(trace_path):
    with open(trace_path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]

def reconcile(repo: Path, trace_path: str, require_mutation: bool) -> int:
    spans = load_spans(trace_path)
    open_ids = [s["span_id"] for s in spans if s.get("status") == "open"]
    if open_ids:
        print("FAIL open spans:", open_ids)
        return 2

    mutating = [
        s for s in spans
        if s.get("op") in WRITE_OPS and s.get("status") == "closed"
    ]
    disk = porcelain_paths(repo)
    span_paths = {s["path"] for s in mutating}
    ghost = sorted(set(disk) - span_paths)
    phantom = sorted(span_paths - set(disk))

    hash_miss = []
    for s in mutating:
        p = repo / s["path"]
        if s["op"] == "write" and sha256_file(p) != s.get("after_hash"):
            hash_miss.append(s["path"])
        elif s["op"] == "delete" and p.exists():
            hash_miss.append(s["path"])

    if require_mutation and not disk and not mutating:
        print("FAIL no-op tree on a mutating task")
        return 3
    if ghost or phantom or hash_miss:
        print("FAIL ghost:", ghost)
        print("FAIL phantom:", phantom)
        print("FAIL hash:", hash_miss)
        return 1
    print("PASS span-tree bijection")
    return 0

if __name__ == "__main__":
    repo = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    trace = sys.argv[2] if len(sys.argv) > 2 else "run.jsonl"
    require = "--require-mutation" in sys.argv
    sys.exit(reconcile(repo, trace, require))

The CI step should stay boring. That is the desired property. After the agent process exits, the job fails on any leftover path. Do not scan the final assistant message for “done” or “fixed.” Those strings are not a working tree.

python reconcile.py . run.jsonl --require-mutation

The following cases are unexecuted examples, not measured production results. They pin the three failure classes the gate is meant to catch.

# test_reconcile_cases.py — unexecuted examples
def test_ghost_write_fails(tmp_repo):
    # an untraced formatter touches extra.py; porcelain has it, spans do not
    assert reconcile(tmp_repo, "run.jsonl", False) == 1

def test_phantom_write_fails(tmp_repo):
    # a closed span claims src/app.py; git is clean
    assert reconcile(tmp_repo, "run.jsonl", False) == 1

def test_noop_on_required_mutation_fails(tmp_repo):
    assert reconcile(tmp_repo, "run.jsonl", True) == 3

Wire the agent so it can only call traced_write, a traced delete, and a traced shell that predeclares output paths. Isolation matters more than the hash function. A laptop already mid-refactor will fail the gate for human reasons: your own dirty files become ghosts. Run the agent in a clean clone. Keep the developer tree out of the ledger.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is a practical place to run that clone-plus-harness loop when the point is to keep the agent's tree off the developer's working copy. The scripts do not depend on it. They run against any model endpoint and any VM. The free options only remove the excuse that isolation needs a dedicated budget before the gate exists.

Limitations are sharp. The harness does not see mutations that never hit git, including ignored paths, Docker volume writes, and database rows. Line-ending conversion and formatters that rerun after the last span will produce hash misses even when the semantic edit is correct; trace the formatter or disable it during the run. Concurrent agents on one working tree break the bijection because porcelain cannot say which span owned a path. Case-insensitive filesystems alias paths the JSONL treats as distinct. Binary assets hash correctly but make traces noisy; keep them out of agent workspaces or accept large JSONL. This control also does not prove the change was right. It proves the change was accounted for. A well-traced, well-hashed, wrong patch still merges if tests are weak.

Do not use this approach for exploratory sessions where the human is supposed to edit the same tree. The gate will fight the human. Do not use it as a secrecy boundary. Hashes of file bytes are not redaction; secrets written into a traced file are still secrets in whatever store keeps the JSONL. Do not use it to rank models. A cheaper model that emits fewer ghost writes is not “better at coding.” It is better at staying inside the adapter. Those are different claims, and mixing them is how generated output gets called engineering.

The durable rule is small. Close every mutating span. Reconcile against porcelain. Fail on leftovers in either set. Until that map is closed, the agent did not finish, no matter how confident the last sentence reads.

Off-Trace Token Counts Cannot Gate a Run

Sam Sun — Wed, 16 Sep 2026 04:52:00 +0000

A usage number that never becomes a span attribute cannot fail a build. It is a caption on a chat UI, and captions do not belong in CI.

Agent runs hide cost in three places at once: the model call, the tool payload, and the retries the panel collapses into one turn. Dashboards mix those numbers after the fact. A trace can keep them on the span that caused them, which is the only representation you can reject in a job without arguing about screenshots.

Think of the vendor usage page as a till receipt you glance at while walking out. The trace is the ledger. If the ledger does not foot, you do not negotiate with the receipt. You fail the run.

This is the slice of loop engineering that stays deterministic when a model writes the glue. Generating the next tool call is cheap. Declaring what that call was allowed to spend is not, unless the spend lives on the span that performed the work.

The schema stays small on purpose. Each span has a kind, a parent pointer, a closed flag, and three integers that default to absent rather than zero. Absent is not free. Zero is a claim that you measured nothing and found nothing.

# trace_budget.py
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from typing import Optional

REQUIRED_KINDS = {"run", "llm", "tool"}


@dataclass(frozen=True)
class Span:
    span_id: str
    parent_id: Optional[str]
    kind: str
    name: str
    input_tokens: Optional[int]
    output_tokens: Optional[int]
    tool_payload_bytes: Optional[int]
    closed: bool


def _opt_int(value, line_no: int) -> Optional[int]:
    if value is None:
        return None
    if isinstance(value, bool) or not isinstance(value, int):
        raise SystemExit(f"line {line_no}: usage fields must be int or null")
    if value < 0:
        raise SystemExit(f"line {line_no}: usage fields must be >= 0")
    return value


def load_jsonl(path: str) -> list[Span]:
    spans: list[Span] = []
    with open(path, encoding="utf-8") as fh:
        for line_no, raw in enumerate(fh, start=1):
            raw = raw.strip()
            if not raw:
                continue
            row = json.loads(raw)
            try:
                parent = row.get("parent_id")
                spans.append(
                    Span(
                        span_id=str(row["span_id"]),
                        parent_id=None if parent in (None, "") else str(parent),
                        kind=str(row["kind"]),
                        name=str(row["name"]),
                        input_tokens=_opt_int(row.get("input_tokens"), line_no),
                        output_tokens=_opt_int(row.get("output_tokens"), line_no),
                        tool_payload_bytes=_opt_int(
                            row.get("tool_payload_bytes"), line_no
                        ),
                        closed=bool(row.get("closed", False)),
                    )
                )
            except KeyError as exc:
                raise SystemExit(f"line {line_no}: missing {exc}") from exc
    return spans

Two rules keep the sums honest. The run root records no tokens and no payload bytes. Only llm and tool leaves do. If the root copies its children, the auditor double-counts, and the ceiling you tuned yesterday becomes a random number tomorrow.

Missing fields stay None. Treating a missing token count as zero makes a broken exporter look efficient. That is how a budget rots: the cheapest-looking run is the one that forgot to emit usage.

class BudgetError(Exception):
    pass


def audit(
    spans: list[Span],
    max_input_tokens: int,
    max_output_tokens: int,
    max_tool_bytes: int,
) -> dict:
    by_id: dict[str, Span] = {}
    for span in spans:
        if span.span_id in by_id:
            raise BudgetError(f"duplicate span_id {span.span_id}")
        if span.kind not in REQUIRED_KINDS:
            raise BudgetError(f"{span.span_id}: unknown kind {span.kind}")
        by_id[span.span_id] = span

    roots = [s for s in spans if s.parent_id is None]
    if len(roots) != 1 or roots[0].kind != "run":
        raise BudgetError("forest must have exactly one run root")
    if roots[0].input_tokens or roots[0].output_tokens or roots[0].tool_payload_bytes:
        raise BudgetError("run root must not carry usage integers")

    for span in spans:
        if span.parent_id is None:
            continue
        if span.parent_id not in by_id:
            raise BudgetError(f"{span.span_id}: missing parent")
        if not span.closed:
            raise BudgetError(f"{span.span_id}: open span")
        if span.kind == "llm" and (
            span.input_tokens is None or span.output_tokens is None
        ):
            raise BudgetError(f"{span.span_id}: llm span missing token attributes")
        if span.kind == "tool" and span.tool_payload_bytes is None:
            raise BudgetError(f"{span.span_id}: tool span missing payload bytes")

    input_tokens = sum(s.input_tokens or 0 for s in spans)
    output_tokens = sum(s.output_tokens or 0 for s in spans)
    tool_bytes = sum(s.tool_payload_bytes or 0 for s in spans)

    if input_tokens > max_input_tokens:
        raise BudgetError(f"input_tokens {input_tokens} > {max_input_tokens}")
    if output_tokens > max_output_tokens:
        raise BudgetError(f"output_tokens {output_tokens} > {max_output_tokens}")
    if tool_bytes > max_tool_bytes:
        raise BudgetError(f"tool_payload_bytes {tool_bytes} > {max_tool_bytes}")

    return {
        "spans": len(spans),
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "tool_payload_bytes": tool_bytes,
    }

The fail cases are mechanical, not stylistic. An open span means the exporter died mid-tool, so the cost is unknown. A missing parent means spend leaked out of the run. A missing integer means you are about to crown a partial trace as a saving. Duplicate span_id values are not a merge. They are a broken primary key.

Calibrate ceilings from traces the process actually wrote. Emit JSONL from the agent, run the auditor, then tighten the integers. A bound copied from a chat panel will miss the first hidden retry, because the panel is allowed to collapse turns and the trace is not.

def main(argv: list[str]) -> int:
    if len(argv) != 5:
        sys.stderr.write(
            "usage: python trace_budget.py RUN.jsonl MAX_IN MAX_OUT MAX_TOOL_BYTES\n"
        )
        return 2
    path, max_in, max_out, max_bytes = argv[1], int(argv[2]), int(argv[3]), int(argv[4])
    try:
        summary = audit(load_jsonl(path), max_in, max_out, max_bytes)
    except BudgetError as exc:
        sys.stderr.write(f"BUDGET {exc}\n")
        return 1
    sys.stdout.write(json.dumps(summary, indent=2) + "\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

A fixture makes the contract visible. The first file is a legal run. The second file drops output_tokens on the llm span. The first command must exit 0. The second must exit 1. If both pass, the auditor is treating absence as zero, and the gate is theater.

{"span_id":"r1","parent_id":null,"kind":"run","name":"issue-triage","closed":true}
{"span_id":"m1","parent_id":"r1","kind":"llm","name":"plan","input_tokens":412,"output_tokens":86,"closed":true}
{"span_id":"t1","parent_id":"r1","kind":"tool","name":"read_file","tool_payload_bytes":1540,"closed":true}
{"span_id":"m2","parent_id":"r1","kind":"llm","name":"answer","input_tokens":903,"output_tokens":140,"closed":true}

python trace_budget.py good.jsonl 2000 400 8000
python trace_budget.py missing_tokens.jsonl 2000 400 8000; echo $?

Retries are where panel counts lie hardest. One user turn in the UI can be two llm spans with the same name after a swallowed timeout. Summing attributes makes that cost visible. A single “tokens used” caption does not, because the caption is allowed to describe the conversation instead of the forest.

Tool payloads need their own integer because they are not tokens. A read_file call can push a hundred kilobytes into the next prompt without a polite usage field from the model vendor. Converting bytes to tokens with a homemade factor will be wrong on binary files, on compressed JSON, and on every encoding edge a demo skips. Keep the units separate. Fail them separately.

Wall-clock duration is a third unit, and it does not belong in this sum. A slow tool on a cold disk can look expensive in time while remaining cheap in tokens. Mixing those into one score produces a number you cannot explain when it moves. If you need latency, put exclusive duration on the span later. Do not smuggle it into a token ceiling.

Generating calibration traces still costs something, even when the gate is local. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding agent with free model access and a free server option, which is enough to emit JSONL spans while you are still choosing ceilings, without pointing the same loop at a paid API. The checker does not depend on that stack. If you already write spans, run it on those.

Pytest pins the fail-closed policy so a later refactor cannot quietly start treating None as zero.

# test_trace_budget.py
import json
from pathlib import Path

import pytest
import trace_budget as tb


def write_jsonl(path: Path, rows: list[dict]) -> None:
    path.write_text("".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8")


def sample_rows() -> list[dict]:
    return [
        {"span_id": "r1", "parent_id": None, "kind": "run", "name": "issue-triage", "closed": True},
        {
            "span_id": "m1",
            "parent_id": "r1",
            "kind": "llm",
            "name": "plan",
            "input_tokens": 412,
            "output_tokens": 86,
            "closed": True,
        },
        {
            "span_id": "t1",
            "parent_id": "r1",
            "kind": "tool",
            "name": "read_file",
            "tool_payload_bytes": 1540,
            "closed": True,
        },
    ]


def test_sum_and_pass(tmp_path: Path) -> None:
    path = tmp_path / "good.jsonl"
    write_jsonl(path, sample_rows())
    summary = tb.audit(tb.load_jsonl(str(path)), 2000, 400, 8000)
    assert summary["input_tokens"] == 412
    assert summary["output_tokens"] == 86
    assert summary["tool_payload_bytes"] == 1540


def test_missing_output_tokens_fails(tmp_path: Path) -> None:
    rows = sample_rows()
    del rows[1]["output_tokens"]
    path = tmp_path / "bad.jsonl"
    write_jsonl(path, rows)
    with pytest.raises(tb.BudgetError, match="missing token"):
        tb.audit(tb.load_jsonl(str(path)), 2000, 400, 8000)


def test_root_usage_is_rejected(tmp_path: Path) -> None:
    rows = sample_rows()
    rows[0]["input_tokens"] = 412
    path = tmp_path / "double.jsonl"
    write_jsonl(path, rows)
    with pytest.raises(tb.BudgetError, match="run root"):
        tb.audit(tb.load_jsonl(str(path)), 2000, 400, 8000)


def test_open_span_fails(tmp_path: Path) -> None:
    rows = sample_rows()
    rows[2]["closed"] = False
    path = tmp_path / "open.jsonl"
    write_jsonl(path, rows)
    with pytest.raises(tb.BudgetError, match="open span"):
        tb.audit(tb.load_jsonl(str(path)), 2000, 400, 8000)

python -m pytest test_trace_budget.py -q

The method is narrow on purpose. It is not a billing system. Provider invoices remain the financial source of truth, because streaming usage is easy to under-count and because cached prompt tokens are not described the same way across vendors. Sampling will also lie. If you drop most tool spans, the sum is a lower bound, not a budget.

Do not put this checker in front of a pipeline that never closes spans, such as a streaming demo that keeps the root open until the socket dies. Do not use the sum as a quality score. A cheap run can still be wrong. Teams that do not emit parent ids should not start here; fix the forest first. Teams that need per-tenant chargeback should not start here either. This auditor has no customer, no discount, and no reserved capacity. It answers one question: did this run stay inside the bound you declared for this job.

Once the gate is in CI, stop reading the chat UI for spend. The UI is still useful for reading the answer. It is not useful for deciding whether the loop is allowed to ship.

A Trace You Cannot Replay Is Only a Log

Sam Sun — Mon, 14 Sep 2026 22:02:27 +0000

Most agent debugging loops are expensive in the worst possible way. You re-run the entire agent to see a bug a second time, paying model tokens, tool calls, and a fresh layer of nondeterminism for the privilege. A recorded trace tells you what happened once. It rarely tells you what will happen after your fix.

That gap is where releases stall. Teams keep hardening their tracing — span cardinality limits, causal ordering, closed-span gates — and still cannot answer the only question blocking the merge: did my change alter the agent's behavior, or did the world move underneath it? The moment a trace can be replayed offline, it stops being a log and becomes a test.

The loop has three parts. Record the boundaries that actually decide behavior, replay them without touching a model endpoint, and treat the first replay miss as your diff.

Three boundaries decide almost everything

Almost every agent bug I have chased came down to a model request, a tool call, or an environment read. Everything else — the span tree, the log lines, the token counters — is commentary on those three. Capture the inputs to those boundaries and their outputs, and you have captured the run.

The trick is keying. A cassette is a map from a canonical hash of the boundary input to the observed output. Hash the message list, the tool schemas, the decoding parameters, and the step index; that key identifies the decision. If the agent reaches the same call with a different prompt, or at a different position in the loop, the key changes and replay fails loudly instead of quietly passing.

Here is a reference implementation. It is compact on purpose, and it is a reference rather than a drop-in: I have not run it against your stack, so treat the shape as the artifact and adapt the edges.

# tape.py — record/replay harness for agent runs (reference implementation)
import hashlib, json, time

class ReplayMiss(RuntimeError):
    """The run asked for a decision that was never recorded."""

def _key(kind: str, payload: dict) -> str:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str)
    return f"{kind}:{hashlib.sha256(blob.encode()).hexdigest()[:16]}"

class Tape:
    def __init__(self, path: str, mode: str = "record"):
        assert mode in ("record", "replay")
        self.mode, self.path = mode, path
        self.queues: dict[str, list] = {}
        if mode == "replay":
            with open(path) as fh:
                for line in fh:
                    rec = json.loads(line)
                    self.queues.setdefault(rec["key"], []).append(rec)
        else:
            self.fh = open(path, "a")

    def call(self, kind: str, payload: dict, do):
        k = _key(kind, payload)
        if self.mode == "record":
            t0 = time.monotonic()
            out = do()
            self.fh.write(json.dumps({
                "key": k,
                "kind": kind,
                "ms": int((time.monotonic() - t0) * 1000),
                "out": out,
            }) + "\n")
            self.fh.flush()
            return out
        queue = self.queues.get(k)
        if not queue:
            raise ReplayMiss(f"{k} was never recorded — the run diverged here")
        return queue.pop(0)["out"]

The agent loop then routes its two expensive calls through the tape and stays otherwise unchanged.

def run(task, tape, client, tools, max_steps=12):
    msgs = [{"role": "user", "content": task}]
    for step in range(1, max_steps + 1):
        schemas = sorted(tools)
        decision = tape.call(
            "llm",
            {"messages": msgs, "tools": schemas, "step": step},
            lambda: client.chat(messages=msgs, tools=schemas),
        )
        if decision["tool"] is None:
            return decision["content"]
        name, args = decision["tool"], decision["args"]
        result = tape.call(
            "tool",
            {"name": name, "args": args, "step": step},
            lambda: tools[name](**args),
        )
        msgs += [
            {"role": "assistant", "content": decision["content"]},
            {"role": "tool", "name": name, "content": json.dumps(result)},
        ]
    raise RuntimeError("step budget exhausted")

Recording a failing run costs exactly one real pass:

$ python -m agent --task "$(cat tasks/issue-412.txt)" --tape runs/412.jsonl --mode record
$ wc -l runs/412.jsonl
17 runs/412.jsonl

Seventeen decisions. That is the whole run, and it now fits in version control next to the failing task. Replay is the same command with a different mode, and no network traffic at all:

$ python -m agent --task "$(cat tasks/issue-412.txt)" --tape runs/412.jsonl --mode replay
ReplayMiss: llm:9f2c41d0a7b3e815 was never recorded — the run diverged here

That exception is the debugging signal. The cassette length tells you how far the run agreed with the recording before it diverged, and the key tells you which input changed. A prompt edit that shifts the first tool call shows up as a miss at step 2. A tool whose schema changed bumps the llm key at step 1. Neither costs a model call to discover.

One consequence of keying on the step index is deliberate strictness: a refactor that produces the same decisions in a different order fails the replay. That is usually what you want before a release, and it is noisy if you are mid-refactor. Keep a separate cassette branch for exploring rather than loosening the key.

What belongs in the cassette, and what must never be

Recording everything is easy and useless. Recording the deciding inputs is a design decision, and it is worth writing down.

Boundary	Capture	Failure it exposes
Model request	message list, sorted tool schemas, decoding params, step index	prompt or schema drift
Tool call	name, args, idempotency key, wall-clock ms	changed arguments, silent retries
Environment	allowlisted keys, values hashed	config drift between machines
Harness	git SHA of the agent loop	cassette recorded under a different code path

Redaction creates a real tension here. Secrets and user data must not land in a cassette, but a prompt that has been scrubbed produces a different key, and a different key means the recording no longer matches the run. The workable compromise is to keep the deciding input intact as a hash field and store a redacted copy beside it for humans to read. Replay uses the hash; you read the copy.

Side effects need the same care. A cassette replays tool calls by returning recorded output, so anything with an irreversible effect — a payment, a deploy, a message send — must be stubbed at the tool layer, not merely recorded. If you cannot stub it, do not route it through the tape; wrap the read path and leave the write path alone.

Limits, and who should skip this

The cassette proves your harness logic, not the model's. Providers can return different text for the same request even with a seed set, so replay is a statement about your code path rather than about inference reproducibility. If the bug lives inside sampling behavior, this loop will not find it, and no amount of tracing will.

Cassettes also rot. Every prompt template edit, tool schema change, or loop restructuring invalidates the recordings that touch it, and the honest response is to re-record rather than to loosen the key. That cost is the price of the guarantee, and it is worth paying only when the failure recurs.

Skip this approach if your runs are short and cheap enough that re-running is free, if your failures are always environmental, or if the agent's whole purpose is long-horizon drift and pinning any single decision would be misleading. A replay harness is for agents with a repeatable entry point, a bounded step budget, and at least one bug that keeps coming back.

Where free capacity changes the arithmetic

The asymmetry in this loop is that recording is expensive and replay is free. One recorded run pays the full model bill once; every later iteration on prompts, tool schemas, and routing logic runs against the cassette with zero tokens and zero tool side effects.

That is where MonkeyCode's free model access fits, and it is a narrow fit rather than a headline. The operator provides free model access and a free server option. The free access lowers the cost of the one pass that has to hit a real endpoint, and it matters most when a cassette rots and needs re-recording. The free server option matters for an adjacent reason: a cassette has to be written on the machine where the tools actually run, because that is the environment holding the credentials, the network path, and the real data shapes. The operator also states a free token allowance, currently described as ten million tokens; quotas and terms are the operator's to define and can change, so confirm them before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the ordering straight. Free access makes the recording pass cheap, and replay is what makes a cheap recording worth having. If you already keep a trace suite for agents, record one cassette for your worst recurring failure before you touch another prompt, then diff the replay instead of the logs.

Call the Agent Fixed Only After a Bounded Trace Diff

Sam Sun — Sun, 13 Sep 2026 20:58:02 +0000

A second agent run that prints a healthy answer is not a fix. The only evidence that a prompt, a tool, or a retry policy changed the system in a controlled way is a bounded diff of spans, tool calls, and logs that still join to a span id.

Stdout is a press release. Traces are the ledger. Most agent failures do not raise. They insert an extra search, rewrite one argument, move a parent, or emit a warning that no longer belongs to any span. If you only compare final text, you accept a different machine and call it the same job.

This article treats the pairwise trace diff as the regression oracle. The artifact is a small recorder, a canonical digest, and a classifier that fails a “green” rerun when the change set is unbounded. It is a proposal you can run locally. It is not a claim about production telemetry you have not measured.

Why the second run lies

An agent loop is closer to a git working tree than to a function. Each tool call is a commit. Each nested span is a branch. Logs are comments that only count if they point at a commit hash. Re-running until the last line looks right is like squashing history because the README matches.

Timestamps make the lie worse. Two runs on a free or loaded endpoint will not share wall clocks, so sorting by time invents order. Causal structure is the parent id, the tool sequence, and the argument digest. Those three survive jitter. Wall time does not.

A useful debug loop therefore stores a failing capture, applies one change, stores a second capture, and asks a narrow question: which trace facts moved, and were those moves on an allowlist? If the answer is “the text got better,” you do not yet have a fix. You have a different story.

A canonical record, not a novel schema

Keep one JSONL file per run. Every line is an event with a kind, a span id, an optional parent, and a payload that can be hashed after sorting keys. The point is joinability. A log line without a span id is orphan evidence. A tool call without a parent is an unrooted commit. Both should fail the capture before you even diff.

# proposal: local capture, not a vendor SDK
from __future__ import annotations

import hashlib, json, time
from dataclasses import dataclass, field
from typing import Any, Iterable

ALLOWED_KINDS = {"span_open", "span_close", "tool", "log"}

@dataclass
class Event:
    kind: str
    span_id: str
    parent_id: str | None
    name: str
    attrs: dict[str, Any] = field(default_factory=dict)
    ts_mono: float = field(default_factory=time.monotonic)

    def validate(self) -> None:
        if self.kind not in ALLOWED_KINDS:
            raise ValueError(f"unknown kind: {self.kind}")
        if not self.span_id:
            raise ValueError("span_id required")
        if self.kind == "log" and self.parent_id is None and not self.span_id:
            raise ValueError("log must join a span")

def _stable(obj: Any) -> str:
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), default=str)

def arg_digest(attrs: dict[str, Any], redact: Iterable[str] = ("api_key", "token", "password")) -> str:
    cleaned = {k: ("<redacted>" if k.lower() in redact else v) for k, v in attrs.items()}
    return hashlib.sha256(_stable(cleaned).encode()).hexdigest()[:16]

def fingerprint(ev: Event) -> tuple:
    # monotonic time is recorded, never used as order
    return (ev.kind, ev.name, ev.span_id, ev.parent_id or "", arg_digest(ev.attrs))

Record in causal order as the loop actually calls tools. Do not sort by ts_mono later and pretend that is history. If a worker interleaves two agent runs, give each run its own file. Mixing forests in one stream is how a “fix” absorbs a neighbor’s spans.

class Capture:
    def __init__(self, run_id: str) -> None:
        self.run_id = run_id
        self.events: list[Event] = []
        self._open: set[str] = set()

    def open_span(self, span_id: str, name: str, parent_id: str | None = None, **attrs: Any) -> None:
        ev = Event("span_open", span_id, parent_id, name, attrs)
        ev.validate()
        self._open.add(span_id)
        self.events.append(ev)

    def tool(self, span_id: str, name: str, parent_id: str, **attrs: Any) -> None:
        if parent_id not in self._open:
            raise ValueError(f"tool {name} cited closed or missing parent {parent_id}")
        ev = Event("tool", span_id, parent_id, name, attrs)
        ev.validate()
        self.events.append(ev)

    def log(self, span_id: str, msg: str, **attrs: Any) -> None:
        if span_id not in self._open:
            raise ValueError(f"orphan log: {msg!r}")
        payload = {"msg": msg, **attrs}
        self.events.append(Event("log", span_id, span_id, "log", payload))

    def close_span(self, span_id: str) -> None:
        if span_id not in self._open:
            raise ValueError(f"double close: {span_id}")
        self._open.remove(span_id)
        self.events.append(Event("span_close", span_id, None, "close", {}))

    def finish(self) -> None:
        if self._open:
            raise ValueError(f"unclosed spans: {sorted(self._open)}")

The finish() gate is intentional. An agent that returns a string while spans remain open has not completed the unit of work you are about to diff. Close first. Diff second. That ordering keeps the second run from looking “better” because it simply dropped cleanup.

Digest, then classify the delta

Hashing the whole JSONL is too brittle. A single log message change would fail every rerun. Diff at the grain you can explain: tool names in order, argument digests, span names with parent edges, and a count of joinable logs. Everything else is noise until you promote it.

from collections import Counter

@dataclass(frozen=True)
class Digest:
    tools: tuple[tuple[str, str], ...]          # (name, arg_digest)
    edges: tuple[tuple[str, str], ...]          # (parent_name, child_name)
    log_join: int
    kinds: tuple[tuple[str, int], ...]

def digest(cap: Capture) -> Digest:
    tools = []
    names = {}
    edges = []
    logs = 0
    for ev in cap.events:
        if ev.kind == "span_open":
            names[ev.span_id] = ev.name
            if ev.parent_id:
                edges.append((names.get(ev.parent_id, ev.parent_id), ev.name))
        elif ev.kind == "tool":
            tools.append((ev.name, arg_digest(ev.attrs)))
        elif ev.kind == "log":
            logs += 1
    kind_counts = tuple(sorted(Counter(ev.kind for ev in cap.events).items()))
    return Digest(tuple(tools), tuple(edges), logs, kind_counts)

@dataclass
class Delta:
    added_tools: list[tuple[str, str]]
    dropped_tools: list[tuple[str, str]]
    arg_drift: list[str]
    edge_churn: list[tuple[str, str]]
    log_delta: int

def diff_traces(before: Digest, after: Digest) -> Delta:
    b_tools, a_tools = list(before.tools), list(after.tools)
    added = [t for t in a_tools if t not in b_tools]
    dropped = [t for t in b_tools if t not in a_tools]
    # same tool name, different digest, same position-ish
    drift = []
    for (n1, d1), (n2, d2) in zip(b_tools, a_tools):
        if n1 == n2 and d1 != d2:
            drift.append(n1)
    churn = [e for e in after.edges if e not in before.edges]
    return Delta(added, dropped, drift, churn, after.log_join - before.log_join)

Call a rerun bounded when every class of change is either empty or explicitly allowed. Added tools are the usual surprise. Argument drift is the quiet one: the model still calls search, but the query picked up a hallucinated filter. Edge churn means the forest rewired even if the tool list looks familiar. Log delta is a weak signal; use it as a smell, not as proof.

ALLOW_ADDED = {("cache_lookup",)}
ALLOW_DRIFT = {"search"}  # only if you stubbed the corpus

def gate_fix(delta: Delta) -> None:
    unexpected_added = [t for t in delta.added_tools if (t[0],) not in ALLOW_ADDED and t[0] not in {a[0] for a in ALLOW_ADDED}]
    if unexpected_added:
        raise AssertionError(f"unbounded add: {unexpected_added}")
    if delta.dropped_tools:
        raise AssertionError(f"dropped tools: {delta.dropped_tools}")
    bad_drift = [n for n in delta.arg_drift if n not in ALLOW_DRIFT]
    if bad_drift:
        raise AssertionError(f"arg drift: {bad_drift}")
    if delta.edge_churn:
        raise AssertionError(f"parent rewrite: {delta.edge_churn}")

Wire that gate after the second capture, not after a human reads stdout. A CI job can store both JSONL files as artifacts. The assertion message is the debug entry point: which tool appeared, which argument hash moved, which parent edge is new. You then open those two files and read only the unmatched events. That is the reusable loop. It does not require a dashboard.

Where a free model path belongs

The expensive part of this method is not the diff. It is obtaining two captures under comparable constraints: same stubbed tools, same redaction, same close-span rule. If inference cost contaminates the experiment, people skip the second capture and declare victory from a single pretty answer.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option. Those two facts are the only product claims here. They matter only as a way to run the failing capture and the candidate capture without folding billed tokens into the comparison. They do not replace stubs, redaction, or the gate above, and they do not imply a particular model name, quota, or hardware profile.

If you already have a stable paid endpoint, keep using it. The oracle is the bounded diff, not the vendor. A free path is useful when the alternative is not measuring the second run at all.

A worked walk through one failure class

Suppose run A calls retrieve then summarize. Run B, after a prompt tweak, calls retrieve, retrieve, then summarize. Final text can look cleaner because the second retrieve stuffed the context. gate_fix should fail on added_tools. The allowlist stays empty until you decide duplicate retrieve is the intended contract. That decision belongs in code, not in a chat thread.

Argument drift is the next class. Run A hashes {"q": "span cardinality"}. Run B hashes {"q": "span cardinality 2024"}. Same tool name, different digest. If your corpus is stubbed, you may allow search drift while you iterate on phrasing. If the tool hits a live index, do not allow it. Live indexes turn a prompt change into an uncontrolled input, and the diff will bounce forever.

Orphan logs are a capture bug, not an agent bug. A print inside a tool that forgot the span id will trip Capture.log. Fix the logger before you debate the model. Otherwise the digest’s log_join count becomes a random integer, and you will start allowing log delta because it is noisy. That is how the oracle decays.

Limitations

This recorder is not OpenTelemetry, not a distributed context propagator, and not a substitute for evaluating answer quality. Hashing sorted JSON will still thrash if tool adapters inject timestamps or request ids into arguments. Strip those fields before arg_digest, or every rerun looks unbounded.

Non-deterministic tools without stubs cannot use a strict tool-sequence gate. You must either freeze the tool, hash a coarser view (name only), or stop claiming the second run is comparable. Streaming token traces are out of scope. So is multi-tenant mixing: one JSONL per run_id, always.

The allowlists are load-bearing. An unbounded ALLOW_DRIFT turns the gate into theater. Review them when the prompt changes, not once per quarter. And never persist raw tool arguments that contain secrets. The digest is supposed to be shareable. The JSONL may not be.

Who should not use this

Do not adopt this loop if your agent is a single deterministic function with no tools. A unit test on the return value is enough. Do not adopt it if you cannot close spans or join logs; the capture will fail closed, which is correct, but you will not get a diff. Do not adopt it as a product tour. If the goal is to try a chat UI, you do not need a trace oracle.

Teams that ship user PII through tool arguments should not store JSONL until redaction is real. The arg_digest helper only covers obvious key names. That is not a compliance program.

The core conclusion does not move. A green rerun is a candidate. It becomes a fix when the trace diff is bounded, the added tools are intended, the arguments that drifted are allowlisted, and every log still names a live span. Until then you only have a nicer last line.

Fail the Run If the Trace Forest Splits

Sam Sun — Sat, 12 Sep 2026 18:57:24 +0000

An agent run that exports more than one trace root did not record a run. It recorded fragments. Those fragments cannot be ordered, cannot be diffed against yesterday’s export, and cannot tell you which tool returned after the parent step had already closed. The process may still have printed a final answer. The observation is already broken.

HTTP services live with forests. A load balancer, a client that never set a propagator, a one-percent sampler — all of them mint extra roots, and on-call still reconstructs a request from logs. Agent loops do not get that luxury. One model step fans into tools. Tools fork. Forks talk to shells, sandboxes, and stdio servers. If traceparent dies at the fork, the collector stores a tree plus a pile of stumps. You will debug the stumps.

This article treats a split forest as a failed run, not as a telemetry warning. The artifact is a small Python runner, a W3C header injector, and a checker you can keep next to the loop. The snippets are labeled examples. They are not production measurements.

Why the forest appears

Imagine a parent span named run that starts a tool named checkout_repo. The tool is a subprocess. Subprocesses do not inherit in-process context objects. They inherit environment variables and file descriptors. If you forget to write TRACEPARENT into that environment, the tool starts a new trace_id. Your backend now holds two traces that share a run_id attribute if you remembered to copy it, or share nothing if you did not.

Logs will still interleave on stdout. They always do. That interleaving is why a log stream cannot be the source of truth for tool order. The forest can, but only when it is a tree.

A second, quieter failure is the late child. The parent step hits a deadline and closes. The subprocess later emits a span that still names the old parent. Some backends attach it. Some drop it. Some create a new root. From the agent’s point of view the tool failed with timeout. From the collector’s point of view a span arrived from a closed world. If you do not name that case, the retry looks like a second successful tool call.

Copied HTTP sampling makes this worse. Edge traffic is high volume and low diagnostic value per span. Agent traffic is the opposite. Dropping the only tool span that mutated state is not “saving cardinality.” It is deleting the receipt.

A contract small enough to test

Three rules fit in one paragraph. One root per run_id. Every tool span must carry a parent that exists in the same trace. A span that starts after its parent ended is not a child; it is either an event named late_tool on that parent, or it is a failure of the run.

That is stricter than typical APM. It should be. You can execute the contract locally with JSONL and no vendor. OpenTelemetry can emit the same shape later. The point is the assertion, not the exporter logo.

Inject context before the tool forks

The runner below mints one trace_id for the run, formats a W3C traceparent, and puts it in the child environment. The child is responsible for starting its body span under that parent. If it ignores the header, it becomes a second root, and the checker fails.

# agent_trace_contract.py — example runner, not a library
from __future__ import annotations

import json, os, secrets, subprocess, sys, time
from dataclasses import asdict, dataclass, field
from typing import Optional

def _hex(n: int) -> str:
    return secrets.token_hex(n)

def format_traceparent(trace_id: str, span_id: str, sampled: bool = True) -> str:
    flags = "01" if sampled else "00"
    return f"00-{trace_id}-{span_id}-{flags}"

def parse_traceparent(value: str) -> tuple[str, str]:
    parts = value.split("-")
    if len(parts) != 4 or parts[0] != "00" or len(parts[1]) != 32 or len(parts[2]) != 16:
        raise ValueError(f"invalid traceparent: {value!r}")
    return parts[1], parts[2]

@dataclass
class Span:
    trace_id: str
    span_id: str
    parent_span_id: Optional[str]
    name: str
    start_ns: int
    end_ns: Optional[int] = None
    status: str = "UNSET"
    attrs: dict = field(default_factory=dict)

    def close(self, status: str = "OK") -> None:
        self.end_ns = time.time_ns()
        self.status = status

class JsonlExporter:
    def __init__(self, path: str) -> None:
        self.path = path
        open(self.path, "w", encoding="utf-8").close()

    def emit(self, span: Span) -> None:
        with open(self.path, "a", encoding="utf-8") as fh:
            fh.write(json.dumps(asdict(span)) + "\n")

class Tracer:
    def __init__(self, exporter: JsonlExporter, run_id: str) -> None:
        self.exporter = exporter
        self.run_id = run_id
        self.trace_id = _hex(16)
        self._stack: list[Span] = []

    def start(self, name: str, attrs: Optional[dict] = None) -> Span:
        parent = self._stack[-1].span_id if self._stack else None
        span = Span(
            trace_id=self.trace_id,
            span_id=_hex(8),
            parent_span_id=parent,
            name=name,
            start_ns=time.time_ns(),
            attrs={"run_id": self.run_id, **(attrs or {})},
        )
        self._stack.append(span)
        return span

    def end(self, span: Span, status: str = "OK") -> None:
        span.close(status)
        if self._stack and self._stack[-1].span_id == span.span_id:
            self._stack.pop()
        self.exporter.emit(span)

def run_tool(tracer: Tracer, argv: list[str], timeout_s: float) -> dict:
    parent = tracer.start("tool:" + os.path.basename(argv[1] if argv[0] == sys.executable else argv[0]))
    env = os.environ.copy()
    env["TRACEPARENT"] = format_traceparent(parent.trace_id, parent.span_id)
    env["RUN_ID"] = tracer.run_id
    env["SPAN_OUT"] = tracer.exporter.path
    try:
        proc = subprocess.run(argv, env=env, capture_output=True, text=True, timeout=timeout_s)
    except subprocess.TimeoutExpired:
        tracer.end(parent, "ERROR")
        return {"ok": False, "error": "timeout"}
    tracer.end(parent, "OK" if proc.returncode == 0 else "ERROR")
    return {"ok": proc.returncode == 0, "stdout": proc.stdout, "stderr": proc.stderr}

The child process is deliberately tiny. It either joins the parent trace or it lies.

# tool_worker.py — example child; pass --orphan to mint a new root
import json, os, sys, time, secrets

def main() -> None:
    out = os.environ["SPAN_OUT"]
    run_id = os.environ.get("RUN_ID", "")
    header = os.environ.get("TRACEPARENT", "")
    orphan = "--orphan" in sys.argv
    if orphan or not header:
        trace_id, parent = secrets.token_hex(16), None
    else:
        parts = header.split("-")
        trace_id, parent = parts[1], parts[2]
    start = time.time_ns()
    time.sleep(0.05)
    record = {
        "trace_id": trace_id,
        "span_id": secrets.token_hex(8),
        "parent_span_id": parent,
        "name": "tool_body:checkout_repo",
        "start_ns": start,
        "end_ns": time.time_ns(),
        "status": "OK",
        "attrs": {"run_id": run_id},
    }
    with open(out, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")
    print(json.dumps({"files": 12}))

if __name__ == "__main__":
    main()

Wire them with a one-shot command. The first invocation should pass. The second should fail the forest check on purpose.

python - <<'PY'
import sys
from agent_trace_contract import JsonlExporter, Tracer, run_tool, load_and_assert
exp = JsonlExporter("/tmp/run.jsonl")
tr = Tracer(exp, run_id="run-demo-1")
root = tr.start("run")
run_tool(tr, [sys.executable, "tool_worker.py"], timeout_s=2.0)
tr.end(root)
print(load_and_assert("/tmp/run.jsonl", "run-demo-1"))
PY

Assert one tree, then diff two trees

The checker is the part that belongs in CI. It does not parse stdout. It does not trust timestamps to rebuild order. It counts roots, distinct trace_ids, parents that point nowhere, and children that start after the parent end_ns.

# same module — example assertion
def load_and_assert(path: str, run_id: str) -> dict:
    with open(path, encoding="utf-8") as fh:
        spans = [json.loads(line) for line in fh if line.strip()]
    mine = [s for s in spans if s.get("attrs", {}).get("run_id") == run_id]
    if not mine:
        raise AssertionError("no spans for run_id")
    ids = {s["span_id"] for s in mine}
    roots = [s for s in mine if not s.get("parent_span_id")]
    missing = [s for s in mine if s.get("parent_span_id") and s["parent_span_id"] not in ids]
    by_id = {s["span_id"]: s for s in mine}
    late = []
    for s in mine:
        parent = by_id.get(s.get("parent_span_id") or "")
        if parent and parent.get("end_ns") and s["start_ns"] > parent["end_ns"]:
            late.append(s["span_id"])
    report = {
        "roots": len(roots),
        "trace_ids": len({s["trace_id"] for s in mine}),
        "missing_parents": len(missing),
        "late_children": len(late),
        "span_count": len(mine),
    }
    if report["trace_ids"] != 1 or report["roots"] != 1 or missing or late:
        raise AssertionError(f"split forest: {report}")
    return report

def graph_key(spans: list[dict], run_id: str) -> list[tuple]:
    mine = [s for s in spans if s.get("attrs", {}).get("run_id") == run_id]
    by_id = {s["span_id"]: s for s in mine}
    rows = []
    for s in mine:
        parent = by_id.get(s.get("parent_span_id") or "")
        rows.append((s["name"], parent["name"] if parent else None, s["status"]))
    return sorted(rows)

A reusable debug loop follows from those two functions. Export run A. Assert a single root. Change one thing — drop TRACEPARENT, add --orphan, or shrink the tool timeout. Export run B. Diff graph_key(A) against graph_key(B). If the graphs diverge in roots or parent names, stop. Do not open the prompt. Do not grep stdout for “success.” The forest already named the fault.

That loop is cheap enough to run on every fixture. Agent volume is not web traffic. Recording every tool span is the conservative default until a span name explodes into unbounded cardinality, which is a different contract and a different failure.

When the question is whether a split forest came from a missing injector or from a flaky model step, you need a second execution environment that does not change the tool graph. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode here only as that second environment: its free model access and free server option are enough to rerun the same loop and compare two JSONL exports, without turning the debug session into an infra ticket. The checker does not depend on that environment. If you already have a scratch box, keep using it.

What this does not cover

The contract is wrong for ordinary HTTP services. Those systems need sampling, tail-based decisions, and the right to mint extra roots at trust boundaries. Copying “one root or fail” onto a public API will page you for client bugs you do not own.

It is also wrong when you cannot wrap the tool. A third-party SaaS that never returns traceparent, and never accepts one, will always look like a stump unless you create a local client span and treat the remote call as an event. Do not pretend the remote side joined your tree.

JSONL is not a collector. It will not backpressure, deduplicate, or survive a crashed runner. Use it to prove the injector. Then export the same fields through OTLP if you already have a backend. Clock skew still exists; this checker uses parent ids and end_ns on spans you control, not wall clocks from two hosts.

Skip the approach if the job is a batch of independent items with no coordinator. Those runs are supposed to be many roots. Inventing a fake parent so the checker stays green is worse than a forest you understand.

If the forest splits, the run is not green. Repair the injector before you tune the prompt. The model cannot cite a source the tracer never joined.

Stdout Cannot Halt an Agent Loop

Sam Sun — Fri, 11 Sep 2026 16:45:41 +0000

An agent loop that only prints iteration=7 is already unbounded. The process can die, two retries can interleave, and a remote model hop can swallow stdout. If the counter and the halt reason are not attributes on a closed span, you do not have a halt condition. You have a log line that hoped someone would read it.

This is not a style complaint. It is a failure-mode complaint. Agent work is a loop with tools, and the loop is the thing that runs away. HTTP services taught us to sample traces. Agent loops punish that habit. A sampled-away iteration is indistinguishable from an iteration that never ran.

The reusable rule is small. Start a parent span for the run. Open a child span for every iteration. Nest each tool call under that child. Write iteration, budget_remaining, and halt_reason as attributes. Close the child before the next hop. If the child never closes, treat the run as a crash, not as a slow success.

Why the print statement lies

Stdout is a stream. A stream has no identity when two workers share a terminal, when a free remote endpoint truncates the response, or when the process is SIGKILLed after the model returns and before print flushes. The line you wanted is the line you lost.

A span is a record with a start, an end, and a parent. That shape survives a killed child. It also survives a model call that leaves the laptop. The remote hop can still fail. It cannot pretend the iteration never existed if the parent already opened a child with a stable work id.

Think of the loop like a checkout line. The receipt is not the chatter at the register. The receipt is the numbered ticket that still exists after the cashier walks away. Agent traces need that ticket. They do not need a louder log level.

A minimum span schema for one loop

Keep the schema boring. Boring is queryable.

run                name=agent.run           attrs: run_id, goal_hash, max_iters
  iter             name=agent.iter          attrs: iteration, budget_remaining, work_id
    llm            name=agent.llm           attrs: role=plan|act, input_hash, output_hash
    tool           name=agent.tool          attrs: tool, schema_hash, ok, error_class
  end              name=agent.halt          attrs: halt_reason, last_iteration

work_id is the idempotency key for the unit of work, not the retry count. Retries of the same tool input keep the same work_id and increment attempt. That stops a retry storm from looking like progress. schema_hash is a digest of the tool’s argument schema at call time. If the hash changes between iterations, the loop is not looping. It is drifting.

halt_reason is an enum you own: budget_exhausted, tool_error, span_open, schema_drift, done. Anything else is unknown, and unknown is a failed run. Do not encode poetry in that field. Downstream alerts cannot parse poetry.

A local writer you can grep

The code below is a proposed, self-contained writer. It is not a production tracer. It appends JSON lines and refuses to start the next iteration while a child is still open. Label it as a workflow you can run on a laptop, not as a measured benchmark.

# loop_trace.py — proposed local span writer for agent loops
from __future__ import annotations

import hashlib, json, time, uuid
from contextlib import contextmanager
from pathlib import Path
from typing import Any, Iterator

HALT_REASONS = {
    "done", "budget_exhausted", "tool_error",
    "span_open", "schema_drift", "unknown",
}

def _now() -> float:
    return time.time()

def schema_hash(schema: dict) -> str:
    blob = json.dumps(schema, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

class LoopTrace:
    def __init__(self, path: Path, run_id: str, max_iters: int) -> None:
        self.path = path
        self.run_id = run_id
        self.max_iters = max_iters
        self._open_iter: str | None = None
        self.last_schema: dict[str, str] = {}
        path.parent.mkdir(parents=True, exist_ok=True)
        self._emit("agent.run", "start", {"run_id": run_id, "max_iters": max_iters})

    def _emit(self, name: str, event: str, attrs: dict[str, Any]) -> None:
        rec = {
            "ts": _now(),
            "run_id": self.run_id,
            "span_id": uuid.uuid4().hex[:16],
            "parent_open": self._open_iter,
            "name": name,
            "event": event,
            "attrs": attrs,
        }
        with self.path.open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(rec, ensure_ascii=False) + "\n")

    @contextmanager
    def iteration(self, n: int) -> Iterator[dict[str, Any]]:
        if self._open_iter is not None:
            self.halt("span_open", n)
            raise RuntimeError("previous iteration span still open")
        if n > self.max_iters:
            self.halt("budget_exhausted", n)
            raise RuntimeError("iteration exceeds max_iters")
        remaining = self.max_iters - n
        work_id = f"{self.run_id}:{n}"
        self._open_iter = work_id
        self._emit("agent.iter", "start", {
            "iteration": n,
            "budget_remaining": remaining,
            "work_id": work_id,
        })
        try:
            yield {"work_id": work_id, "budget_remaining": remaining}
        finally:
            self._emit("agent.iter", "end", {"work_id": work_id, "iteration": n})
            self._open_iter = None

    def tool(self, name: str, schema: dict, ok: bool, error_class: str | None = None) -> None:
        digest = schema_hash(schema)
        prev = self.last_schema.get(name)
        if prev and prev != digest:
            self.halt("schema_drift", -1)
            raise RuntimeError(f"schema drift for {name}: {prev} -> {digest}")
        self.last_schema[name] = digest
        self._emit("agent.tool", "event", {
            "tool": name,
            "schema_hash": digest,
            "ok": ok,
            "error_class": error_class,
        })
        if not ok:
            self.halt("tool_error", -1)

    def halt(self, reason: str, last_iteration: int) -> None:
        if reason not in HALT_REASONS:
            reason = "unknown"
        self._emit("agent.halt", "end", {
            "halt_reason": reason,
            "last_iteration": last_iteration,
            "span_still_open": self._open_iter is not None,
        })

A thin loop around that writer makes the policy visible. The model call is a function you inject. Keep the injection local so a remote hop cannot close your spans for you.

# run_loop.py — proposed control loop, not an executed benchmark
from pathlib import Path
from loop_trace import LoopTrace

MAX_ITERS = 8
TOOL_SCHEMA = {"type": "object", "properties": {"path": {"type": "string"}}}

def call_model(prompt: str) -> dict:
    # Replace with your client. The span must start and end around this call.
    return {"action": "tool", "tool": "read_file", "ok": True}

def main() -> None:
    trace = LoopTrace(Path("runs/latest.jsonl"), run_id="run-20260911", max_iters=MAX_ITERS)
    halt = "unknown"
    last = 0
    try:
        for n in range(1, MAX_ITERS + 1):
            last = n
            with trace.iteration(n):
                decision = call_model(f"iter={n}")
                if decision["action"] == "stop":
                    halt = "done"
                    break
                trace.tool(decision["tool"], TOOL_SCHEMA, ok=bool(decision["ok"]))
        else:
            halt = "budget_exhausted"
    except RuntimeError as exc:
        halt = "span_open" if "open" in str(exc) else "unknown"
        raise
    finally:
        trace.halt(halt, last)

if __name__ == "__main__":
    main()

The interesting line is not the model client. It is with trace.iteration(n). That context manager is the cap. If call_model blocks forever, the child stays open, and the next process that reads the JSONL can see an agent.iter start with no matching end. That is a cheaper crash signal than a wall-clock timeout you invented after the fact.

Query the run, do not tail the terminal

Once the file exists, the debug loop is a query. No dashboard required.

python run_loop.py

jq -c 'select(.name=="agent.iter") | {event, iteration:.attrs.iteration, work:.attrs.work_id}' runs/latest.jsonl

jq -c 'select(.name=="agent.halt") | .attrs' runs/latest.jsonl

# unpaired starts: iterations that never closed
jq -s '
  map(select(.name=="agent.iter"))
  | group_by(.attrs.work_id)
  | map(select(map(.event)|index("end")|not))
  | .[] | .[0].attrs
' runs/latest.jsonl

A proposed regression check belongs next to the writer. It does not prove production behavior. It proves the writer refuses a second start.

# test_loop_trace.py — proposed tests
from pathlib import Path
from loop_trace import LoopTrace
import pytest

def test_open_child_blocks_next_iter(tmp_path: Path) -> None:
    t = LoopTrace(tmp_path / "r.jsonl", "r1", max_iters=3)
    cm = t.iteration(1)
    cm.__enter__()
    with pytest.raises(RuntimeError, match="still open"):
        with t.iteration(2):
            pass

def test_schema_drift_halts(tmp_path: Path) -> None:
    t = LoopTrace(tmp_path / "r.jsonl", "r2", max_iters=3)
    with t.iteration(1):
        t.tool("read_file", {"properties": {"path": {"type": "string"}}}, ok=True)
    with t.iteration(2):
        with pytest.raises(RuntimeError, match="schema drift"):
            t.tool("read_file", {"properties": {"id": {"type": "integer"}}}, ok=True)

Run them with pytest -q test_loop_trace.py. If the first test ever goes green while a child is open, the rest of the workflow is theater.

The remote hop is why this has to be local

Agent loops often send the planning step off-box. That is the moment stdout becomes fiction. The laptop still owns the iteration. The remote endpoint owns a completion. Mix those clocks and you will debug the network instead of the loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough to exercise the remote hop without standing up a private GPU box. Use that hop as the call_model body. Do not let it own halt_reason. The writer above starts and ends on your side of the request. The free server can return a tool decision. It cannot close a span it never opened.

Keep payloads out of the JSONL if the goal text is sensitive. Store hashes, not prompts. The earlier data-boundary problem still exists; this article does not re-solve it. It only keeps the loop counter from riding along with the prompt into a remote log you do not control.

A practical split looks like this. Local process: open agent.iter, hash the prompt, call the remote model, record output_hash, close agent.iter. Remote process: answer the completion. If the free server is also where you park a tiny collector, ship the JSONL after the halt span, not during the tool call. Export during the call is how you duplicate bodies and blow a quiet size cap you did not measure.

What this does not do

It does not replace OpenTelemetry. If your team already emits OTLP with a real collector, map iteration and halt_reason onto that pipeline and delete this file. It does not give you tail sampling. Every iteration is written because a missing iteration is the bug. If a run can do thousands of cheap tool calls, this writer will grow a large JSONL. Cap max_iters first. Do not “fix” size by dropping ends.

It does not prove the model was right. A halt_reason=done span means the loop stopped. It does not mean the tool did the useful thing. Pair this with a closed-span success gate if you need that check. This piece only makes the counter survive.

Skip this approach if the agent is a single model call with no tools. Skip it if you cannot write to disk on the machine that starts the loop. Skip it for multi-tenant agents that put raw user text into span attributes. Hash or drop those fields. The writer will not redact them for you.

Time-sensitive product numbers change. This workflow does not depend on a published token quota, a hardware SKU, or a durability promise. It depends on one closed child per iteration. If you cannot attest that, the loop is still printing.

The core test is one command after a crash: jq 'select(.name=="agent.halt")' runs/latest.jsonl. Empty output means the run never halted. That is the bug. Fix the writer before you tune the prompt.

Treat Span Cardinality as a Hard Error

Sam Sun — Wed, 09 Sep 2026 16:45:18 +0000

Agent traces usually fail as landfills, not as black boxes. The run emitted spans. Too many of them. Each tool call copied the prompt, the raw JSON, and a pile of metadata that no later diff will read. The debug loop then drowns in its own evidence.

Cardinality is the failure mode that looks like success. Status is OK. Parent ids exist. Timestamps are monotonic enough to sort. The trace is still useless, because the tree grew faster than anyone can compare. A free endpoint does not change that arithmetic. It only makes it cheaper to grow the landfill.

Think of a suitcase and a stack of store receipts. The itinerary tells you which cities you visited and in which order. The receipts prove a purchase happened, but stuffing every receipt into the bag does not make the itinerary clearer. Agent traces need the itinerary. They rarely need the full receipt. Hash the receipt. Keep the route.

The practical rule is simple. Treat span cardinality as a hard error, the same way you treat an unclosed file handle. If a run exceeds a byte budget, a child-fan-out cap, or a payload-shape contract, fail the run before you archive it. A huge green trace is not a green run. It is an unbounded write.

This is adjacent to logging culture, not a replacement for it. Logs can be verbose on a laptop. A shared debug loop cannot. Once two runs must be compared, extra attributes stop being context and start being noise. The comparison key has to be small, stable, and reconstructible from the span tree rather than from prose buried in a blob.

A usable span for a tool call is narrow. Name, parent id, status, duration, a canonical payload hash, and a schema fingerprint of sorted keys. That is enough to ask whether this run called the same tools, with the same shapes, in the same tree. Full prompts belong behind a redact-then-store gate, not in the default export.

Suppose eight child tools each carry a 4KB body. That is 32KB before attributes. Ten turns become 320KB. Retries double it. The model then echoes prior tool results into the next prompt, and the next span copies that echo again. Nothing in that sequence looks like a 429. The server still fills. Diffs still lie, because two runs that did the same work hash differently once a timestamp or a request id leaks into the body.

Canonicalization is the fix, not sampling. Sampling drops the failure path you came for. Canonical JSON with sorted keys, then SHA-256, keeps the comparison and throws away the noise. If a field is allowed to change between runs, strip it before the hash. If it is not allowed to change, leave it in and let the hash break. That break is the signal.

The harness below is a local, unexecuted example. It does not report production timings or vendor quotas. Run it with Python 3.11+. It encodes three gates: a total attribute-byte budget, a maximum number of children per parent, and a requirement that every tool.call span has a matching tool.result child. Fail any gate and the trace is not archived.

# span_budget.py
from __future__ import annotations

import hashlib
import json
import unittest
from dataclasses import dataclass, field
from typing import Any, Iterable

DROP_KEYS = frozenset({"ts", "timestamp", "request_id", "trace_id", "span_id"})
MAX_ATTR_BYTES = 32_768
MAX_CHILDREN = 8


def canonical(value: Any) -> Any:
    if isinstance(value, dict):
        return {
            str(k): canonical(v)
            for k, v in sorted(value.items(), key=lambda kv: str(kv[0]))
            if str(k) not in DROP_KEYS
        }
    if isinstance(value, list):
        return [canonical(v) for v in value]
    return value


def payload_hash(payload: Any) -> str:
    blob = json.dumps(canonical(payload), separators=(",", ":"), ensure_ascii=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()


def schema_fingerprint(payload: Any) -> str:
    if not isinstance(payload, dict):
        return type(payload).__name__
    return ",".join(sorted(str(k) for k in payload.keys()))


@dataclass
class Span:
    span_id: str
    parent_id: str | None
    name: str
    kind: str
    status: str
    payload: dict[str, Any] = field(default_factory=dict)

    def compact(self) -> dict[str, Any]:
        return {
            "span_id": self.span_id,
            "parent_id": self.parent_id,
            "name": self.name,
            "kind": self.kind,
            "status": self.status,
            "schema": schema_fingerprint(self.payload),
            "payload_sha256": payload_hash(self.payload),
        }


class SpanBudgetError(RuntimeError):
    pass


def attr_bytes(spans: Iterable[Span]) -> int:
    return sum(len(json.dumps(s.compact(), separators=(",", ":")).encode()) for s in spans)


def enforce_budget(spans: list[Span]) -> list[dict[str, Any]]:
    children: dict[str, int] = {}
    calls = {s.span_id: s for s in spans if s.kind == "tool.call"}
    results_by_parent = {s.parent_id for s in spans if s.kind == "tool.result"}

    for span in spans:
        if span.parent_id:
            children[span.parent_id] = children.get(span.parent_id, 0) + 1
            if children[span.parent_id] > MAX_CHILDREN:
                raise SpanBudgetError(
                    f"fan-out {children[span.parent_id]} at parent {span.parent_id}"
                )

    missing = [sid for sid in calls if sid not in results_by_parent]
    if missing:
        raise SpanBudgetError(f"tool.call missing tool.result: {missing}")

    size = attr_bytes(spans)
    if size > MAX_ATTR_BYTES:
        raise SpanBudgetError(f"compact trace is {size} bytes; cap is {MAX_ATTR_BYTES}")

    return [s.compact() for s in spans]


def tree_hash(compact_spans: list[dict[str, Any]]) -> str:
    ordered = sorted(compact_spans, key=lambda s: (s["parent_id"] or "", s["name"], s["span_id"]))
    return payload_hash(ordered)


class SpanBudgetTests(unittest.TestCase):
    def test_hash_ignores_volatile_fields(self):
        a = {"query": "orders", "ts": "2026-09-09T00:00:00Z", "request_id": "r1"}
        b = {"request_id": "r2", "query": "orders", "ts": "2026-09-09T00:00:01Z"}
        self.assertEqual(payload_hash(a), payload_hash(b))

    def test_orphan_tool_call_fails(self):
        spans = [
            Span("root", None, "agent.run", "run", "ok"),
            Span("c1", "root", "sql.query", "tool.call", "ok", {"sql": "select 1"}),
        ]
        with self.assertRaises(SpanBudgetError):
            enforce_budget(spans)

    def test_matching_result_compacts(self):
        spans = [
            Span("root", None, "agent.run", "run", "ok"),
            Span("c1", "root", "sql.query", "tool.call", "ok", {"sql": "select 1"}),
            Span("r1", "c1", "sql.query", "tool.result", "ok", {"rows": 1}),
        ]
        compact = enforce_budget(spans)
        self.assertEqual(len(compact), 3)
        self.assertEqual(len(tree_hash(compact)), 64)


if __name__ == "__main__":
    unittest.main()

Run it as a gate, not as a viewer.

python span_budget.py

Wire the same function at the end of an agent turn. Compact first. Hash the tree. Persist only the compact list. If you need the raw payload later, write it to a side channel that is off by default and keyed by payload_sha256. The debug loop then diffs tree hashes. A changed hash with an unchanged schema fingerprint points at data. A changed fingerprint points at contract drift. A budget error points at the agent growing a bush instead of a path.

That loop is where a free model endpoint earns its keep. Re-running the agent to see whether the tree hash is stable should not require a paid inference budget on day one. A free server is relevant for the same reason: the compact export is small enough to keep next to the process, and the raw landfill is not. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option fit this workflow only as an iteration surface. They do not remove the budget. They do not define span semantics. If the compact trace cannot be explained without the product name, the trace is still too large.

Do not confuse this with sampling ratios copied from request-centric APM. An agent run is one business action with a fan-out of tools. Dropping 90% of spans drops the tool that mattered. A hard cap on children is stricter and more honest. If the agent needed nine tools, the budget should fail in CI, not silently omit the ninth in production.

Limitations are real. Hashing destroys ad-hoc grep. If a legal process requires the literal prompt, this default export will not satisfy it; keep a locked raw channel with a retention clock, and do not pretend the compact trace is the record. Schema fingerprints also lie when two payloads share keys but not types. Extend schema_fingerprint to walk types if your tools return unions. The byte cap is a constant, not a discovered optimum. Raise it only when a failing test names the span that would not fit.

Skip this approach if the agent is a single function call with no tools. Skip it if every span is already under a few hundred bytes and two runs already diff cleanly. Skip it if you cannot strip volatile keys, because the hash will flap and the team will disable the gate. A disabled cardinality gate is worse than no gate. It trains people to ignore the red bar.

The core conclusion does not depend on a vendor. Archive the itinerary. Hash the receipts. Fail the run when the suitcase will not close. If a free server is already idle, run python span_budget.py before adding another raw log line to a tool span.

Causal Order Beats Timestamps in Agent Traces

Sam Sun — Mon, 07 Sep 2026 14:33:57 +0000

Wall-clock order is not causal order. If you sort an agent run by timestamp on a shared or free endpoint, a tool span can appear to start before the prompt that issued it, and a retry can look like the original call. Debugging then becomes a study of the logger, not the agent. The reusable fix is a happens-before contract: every tool names its parent prompt, every retry names the failed attempt, and a test fails the run when those edges are missing or cyclic.

This is a data-model problem, not a dashboard theme. It shows up the moment two requests share a queue.

Why clocks invert on contended runtimes

A laptop makes timestamps feel trustworthy. The model returns, the tool runs, the next prompt starts, and the file order matches the work. Shared inference is different. A crowded collector can delay flush, batch spans, or stamp arrival time instead of produce time. The numbers still look precise. They are precise about arrival, not about cause.

Think of a station board that sorts train cars by when a clerk stamped the paper, not by which coupler joined them. You get a clean sequence. You cannot rebuild the train.

Agent loops make the inversion worse. A retry often gets a new span id and a fresh timestamp that lands between two children of the original prompt. Sorted views then show a tool from attempt one answering a prompt from attempt two. The summary looks decisive. The graph is wrong.

Recent developer talk about agents keeps circling a related failure: systems that assume. Timestamp sort is one of those assumptions. It survives review because every field is populated. It fails only when you ask whether A could have caused B.

A span record that carries cause

Do not start with a vendor schema. Start with four facts you can test. A span needs an id, an optional parent, a kind, and an explicit caused_by list. Timestamps stay as diagnostics. They do not decide order.

The record below is a proposal you can paste into a test file. It is not a production telemetry standard.

# causal_spans.py
from __future__ import annotations

from dataclasses import dataclass
from typing import Optional

ALLOWED_KINDS = frozenset({"prompt", "tool", "retry", "summary"})


@dataclass(frozen=True)
class Span:
    span_id: str
    parent_id: Optional[str]
    kind: str
    name: str
    t_start_ms: int
    t_end_ms: int
    mutates: tuple[str, ...] = ()
    caused_by: tuple[str, ...] = ()

    def __post_init__(self) -> None:
        if self.kind not in ALLOWED_KINDS:
            raise ValueError(f"unknown kind: {self.kind}")
        if not self.span_id:
            raise ValueError("span_id is empty")
        if self.t_end_ms < self.t_start_ms:
            raise ValueError(f"{self.span_id} ends before it starts")

The mutates field is the other half of the story. Two tool spans may share a prompt parent and still conflict if they write the same path. Causal order is not only what triggered what. It is also what may not overlap.

Rules a debug loop can fail closed on

A validator should reject a run for missing edges before anyone reads a waterfall. Four checks cover the failure without pretending to be a full tracing product.

Parent integrity comes first. A non-root span must point at an id that exists in the same run. Orphans are not interesting unknowns. They are broken export.

Kind edges come next. A tool span must be caused_by a prompt or a retry. A retry must be caused_by the prompt or tool it replaces. A summary must name every prompt that contributed to it. If the summary cannot name those prompts, it is a caption, not a trace.

Acyclicity is the third cut. caused_by must form a DAG. Clock skew cannot be allowed to hide a cycle by making the later node look earlier.

Mutation exclusion is the last cut. If two tool spans share an ancestor prompt and their mutates sets intersect, they must be ordered by an explicit edge. Parallel read-only tools may overlap. Parallel writers to the same file may not.

Notice what is absent. There is no success boolean on the span. Completeness of edges is a different question from success of the task. Mixing them produces green runs with impossible histories.

A failing test you can keep

The example below is an unexecuted template. Point it at traces you already store. It does not need a vendor SDK.

# test_causal_spans.py
from collections import defaultdict, deque
from typing import Dict, List

from causal_spans import Span


class CausalTraceError(AssertionError):
    pass


def index_spans(spans: List[Span]) -> Dict[str, Span]:
    ids = [s.span_id for s in spans]
    if len(ids) != len(set(ids)):
        raise CausalTraceError("duplicate span_id")
    return {s.span_id: s for s in spans}


def assert_parents_exist(by_id: Dict[str, Span]) -> None:
    for span in by_id.values():
        if span.parent_id and span.parent_id not in by_id:
            raise CausalTraceError(f"{span.span_id} parent missing")
        for cause in span.caused_by:
            if cause not in by_id:
                raise CausalTraceError(f"{span.span_id} cause {cause} missing")


def assert_kind_edges(by_id: Dict[str, Span]) -> None:
    for span in by_id.values():
        causes = [by_id[c] for c in span.caused_by]
        if span.kind == "tool" and not any(
            c.kind in {"prompt", "retry"} for c in causes
        ):
            raise CausalTraceError(f"tool {span.span_id} lacks prompt cause")
        if span.kind == "retry" and not any(
            c.kind in {"prompt", "tool"} for c in causes
        ):
            raise CausalTraceError(f"retry {span.span_id} lacks origin")


def assert_acyclic(by_id: Dict[str, Span]) -> None:
    indeg = {sid: 0 for sid in by_id}
    children: dict[str, list[str]] = defaultdict(list)
    for span in by_id.values():
        edges = list(span.caused_by)
        if span.parent_id:
            edges.append(span.parent_id)
        for src in edges:
            children[src].append(span.span_id)
            indeg[span.span_id] += 1
    q = deque([sid for sid, d in indeg.items() if d == 0])
    seen = 0
    while q:
        node = q.popleft()
        seen += 1
        for nxt in children[node]:
            indeg[nxt] -= 1
            if indeg[nxt] == 0:
                q.append(nxt)
    if seen != len(by_id):
        raise CausalTraceError("caused_by graph has a cycle")


def assert_writer_order(by_id: Dict[str, Span]) -> None:
    writers: dict[str, list[Span]] = defaultdict(list)
    for span in by_id.values():
        if span.kind != "tool":
            continue
        for path in span.mutates:
            writers[path].append(span)
    for path, group in writers.items():
        if len(group) < 2:
            continue
        ids = {s.span_id for s in group}
        for span in group:
            others = ids - {span.span_id}
            linked = set(span.caused_by) | ({span.parent_id} if span.parent_id else set())
            if others and not (others & linked):
                raise CausalTraceError(f"unordered writers on {path}")


def validate_run(spans: List[Span]) -> None:
    by_id = index_spans(spans)
    assert_parents_exist(by_id)
    assert_kind_edges(by_id)
    assert_acyclic(by_id)
    assert_writer_order(by_id)


def test_clock_skew_does_not_reorder_cause():
    prompt = Span("p1", None, "prompt", "plan", 1000, 1400)
    tool = Span(
        "t1",
        "p1",
        "tool",
        "edit",
        900,  # collector stamped this earlier than the prompt
        1100,
        mutates=("src/app.py",),
        caused_by=("p1",),
    )
    validate_run([prompt, tool])  # must pass: cause is explicit


def test_retry_without_origin_fails():
    prompt = Span("p1", None, "prompt", "plan", 1000, 1200)
    retry = Span("r1", "p1", "retry", "plan", 1300, 1500, caused_by=())
    try:
        validate_run([prompt, retry])
        raise AssertionError("expected CausalTraceError")
    except CausalTraceError:
        pass

Run the file with a boring command.

python -m pytest test_causal_spans.py -q

If your exporter already writes JSON, inspect cardinality before you argue about latency.

python - <<'PY'
import json, pathlib
raw = json.loads(pathlib.Path("run.json").read_text())
spans = raw["spans"]
print(len(spans), "spans")
print("kinds", sorted({s["kind"] for s in spans}))
print("missing caused_by", sum(1 for s in spans if s["kind"] != "prompt" and not s.get("caused_by")))
PY

The first test is the point. The tool's t_start_ms is earlier than the prompt. A waterfall sorted by clock would draw the tool on the left. The validator still accepts the run because caused_by names p1. The second test rejects a retry that only has a parent pointer. Parent is structure. Cause is history. You need both.

How this changes the debug loop

A reusable loop does not start in the model output. It starts by assembling the span list for one run id, running validate_run, and only then reading names. If validation fails, the human job is to repair export, not to interpret the agent. That split sounds petty until a night of "the tool hallucinated a path" turns out to be two writers with overlapping clocks.

After the graph is legal, diffs belong on the mutated paths, not on the concatenated log. Compare the mutates tuple to git diff -- path or to a working-tree snapshot keyed by span_id. A tool span that mutates nothing should not appear in a code-change review. A tool span that mutates a path with no causal edge should not appear either.

Retries are the usual source of duplicated narrative. Keep the failed tool span. Add a retry span that lists the failed tool in caused_by. Do not delete the first attempt to make the waterfall pretty. Deletion is how teams lose the only evidence that the second attempt was not the first.

A small shell check keeps the habit cheap.

git diff --name-only HEAD | sort > /tmp/changed.txt
python - <<'PY'
import json, pathlib
mut = set()
for s in json.loads(pathlib.Path("run.json").read_text())["spans"]:
    if s.get("kind") == "tool":
        mut.update(s.get("mutates") or [])
changed = set(pathlib.Path("/tmp/changed.txt").read_text().split())
print("traced but not in git", sorted(mut - changed))
print("in git but not traced", sorted(changed - mut))
PY

None of this requires a particular vendor. It requires that whatever emits spans can add three fields: kind, caused_by, mutates. If an exporter cannot, treat its timestamps as comments.

Where a free model path fits, and where it does not

Shared free inference is a good generator of the bug this article is about. Queue delay and batched logs show up there first. They also show up on busy paid clusters, which is why the tests are not a free-tier workaround. They are a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can emit agent runs that you then validate locally with the snippet above. Keep the validator next to your tests, not next to the model. If the server is gone tomorrow, the happens-before rules remain.

Do not send prompt bodies to a shared collector if the run includes secrets. This article is about order, not about data boundaries. Keep stored fields small: ids, kinds, paths, edges. Leave payload off the trace you keep for CI.

Limitations, and who should skip this

The DAG check does not prove the agent was right. It proves the story of the run is internally possible. A legal graph can still encode a bad plan.

The writer rule is conservative. It will flag parallel formatters that touch the same file even when the operations commute. If your tools are proven commutative, add an allow-list of names rather than deleting the check.

Clock fields remain useful for latency once cause is known. Computing a distribution on t_end_ms - t_start_ms for tool spans is fine after validate_run passes. Doing it before is how a skew outlier becomes a performance myth.

Skip this approach if you run a single synchronous agent on one machine with one writer and you already block on each tool. Your wall clock is probably causal, and the extra schema is noise. Also skip it if you cannot add fields to the exporter. Wrapping broken traces in a prettier test only produces prettier fiction.

If you already dump spans from a local loop, point the tests at one free-server run and one laptop run and compare which edges go missing. That difference is the observability gap. The model choice is secondary.

Gate Agent Success on Closed Spans

Sam Sun — Mon, 07 Sep 2026 03:52:25 +0000

The last assistant token is not a status code. If any tool span is still open, the run failed, even when the model wrote a confident wrap-up.

Most agent UIs collapse that distinction. They paint a completed bubble as soon as the model stream ends. The trace underneath is often uglier: a read_file span that started, an apply_patch span that never returned, and a final sentence that assumes both finished. Treating the prose as success is how silent partial work lands on a branch.

This is the same class of bug as an HTTP 200 from a checkout handler while the payment span is still in flight. The outer message completed. The inner contract did not.

What closed means here

A usable agent trace is a small DAG, not a log tail. Each model turn and each tool call is a span with a stable id, a parent id, a start timestamp, and an end timestamp. Status is one of ok, error, or timeout. Missing status is not a fourth kind of success. It is incomplete.

A run is complete only when every span that started has an end, every tool id cited by a model span exists as a tool span, and the root status is derived from children rather than copied from the last token. Incomplete is a failure class of its own. It is not "unknown" and it is not "probably fine."

Free model endpoints and shared free servers make this class common. Idle cuts, process preemption, and dropped streams tend to kill the tool side first and leave the chat side looking finished. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter here only as a place you can persist those traces; they do not relax the contract. If spans never hit disk, you cannot gate on them.

A proposed checker

The validator below is a proposed gate, not a claim about a private production fleet. It has not been executed against a live workload in this article. Point it at JSON you already collect. The schema is small on purpose so a thin wrapper around any runtime can emit it.

{
  "run_id": "run_7f3a",
  "root_id": "span_root",
  "spans": [
    {
      "id": "span_root",
      "parent_id": null,
      "kind": "run",
      "name": "agent_turn",
      "started_at": "2026-09-06T10:01:00.000Z",
      "ended_at": "2026-09-06T10:01:08.410Z",
      "status": "ok",
      "attrs": {}
    },
    {
      "id": "span_m1",
      "parent_id": "span_root",
      "kind": "model",
      "name": "plan",
      "started_at": "2026-09-06T10:01:00.020Z",
      "ended_at": "2026-09-06T10:01:02.100Z",
      "status": "ok",
      "attrs": {"cites_tool_ids": ["span_t1"]}
    },
    {
      "id": "span_t1",
      "parent_id": "span_root",
      "kind": "tool",
      "name": "apply_patch",
      "started_at": "2026-09-06T10:01:02.120Z",
      "ended_at": null,
      "status": null,
      "attrs": {"tool_call_id": "call_9"}
    }
  ]
}

That sample is a failure. The root claims ok. The model span cites span_t1. The tool span never ended. A human reading the final message would miss it. A checker will not.

#!/usr/bin/env python3
"""Proposed agent-trace completeness gate. Unexecuted example."""
from __future__ import annotations

import json
import sys
from hashlib import sha256
from pathlib import Path
from typing import Any

ALLOWED_STATUS = {"ok", "error", "timeout"}


def load_trace(path: Path) -> dict[str, Any]:
    data = json.loads(path.read_text())
    if not isinstance(data.get("spans"), list) or not data.get("run_id"):
        raise ValueError("trace must include run_id and spans[]")
    return data


def issues_for(trace: dict[str, Any]) -> list[str]:
    spans = {s["id"]: s for s in trace["spans"] if "id" in s}
    problems: list[str] = []
    if trace.get("root_id") not in spans:
        problems.append("missing_root")
        return problems

    for span in spans.values():
        if not span.get("started_at"):
            problems.append(f"no_start:{span['id']}")
        ended = span.get("ended_at")
        status = span.get("status")
        if ended is None or status not in ALLOWED_STATUS:
            problems.append(
                f"open_span:{span['id']}:{span.get('kind')}:{span.get('name')}"
            )
        parent = span.get("parent_id")
        if parent and parent not in spans:
            problems.append(f"orphan_parent:{span['id']}")

    for span in spans.values():
        if span.get("kind") != "model":
            continue
        cited = span.get("attrs", {}).get("cites_tool_ids", [])
        for tool_id in cited:
            child = spans.get(tool_id)
            if not child or child.get("kind") != "tool":
                problems.append(f"cited_missing_tool:{span['id']}->{tool_id}")

    root = spans[trace["root_id"]]
    child_bad = any(
        s.get("status") in {"error", "timeout"} or s.get("ended_at") is None
        for s in spans.values()
        if s.get("id") != root["id"]
    )
    if child_bad and root.get("status") == "ok":
        problems.append("root_ok_with_incomplete_or_failed_child")
    return problems


def fingerprint(problems: list[str]) -> str:
    blob = "|".join(sorted(problems)).encode()
    return sha256(blob).hexdigest()[:12]


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print("usage: check_trace.py TRACE.json", file=sys.stderr)
        return 2
    trace = load_trace(Path(argv[1]))
    problems = issues_for(trace)
    result = {
        "run_id": trace["run_id"],
        "complete": not problems,
        "problem_count": len(problems),
        "problems": problems,
        "fingerprint": fingerprint(problems) if problems else None,
    }
    print(json.dumps(result, indent=2))
    return 0 if result["complete"] else 1


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Wire it as a step that can fail the job. Dashboards are optional. The process exit code is not.

python3 check_trace.py traces/run_7f3a.json
echo $?
# expected for the sample: 1

Exit code 1 means do not merge, do not apply the patch, do not send the email the agent drafted. Exit code 0 means the trace is structurally closed. It still does not mean the patch is correct.

The fingerprint is the part that scales. Incomplete runs cluster. open_span:span_t1:tool:apply_patch plus root_ok_with_incomplete_or_failed_child will hash the same way across nights if the same tool is the one that never returns. Debug the cluster. Do not debug each cheerful wrap-up as a unique prompt failure.

A tiny pytest wrapper makes the same rule visible next to unit tests. Label it as a contract test on fixtures, not as evidence that a hosted agent passed.

# proposed test; fixtures live under traces/
from pathlib import Path
from check_trace import issues_for, load_trace


def test_open_tool_span_fails_the_run():
    trace = load_trace(Path("traces/run_7f3a.json"))
    problems = issues_for(trace)
    assert "root_ok_with_incomplete_or_failed_child" in problems
    assert any(p.startswith("open_span:") for p in problems)

Emit the end event locally

When you wrap a tool, the local process is the source of truth for span closure, not the model. The model can still emit "done" after you kill the child. Record timeout yourself.

# proposed wrapper sketch; timestamps come from the wrapper clock
import subprocess, time
from datetime import datetime, timezone


def utcnow():
    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")


def run_tool(span: dict, argv: list[str], timeout_s: float) -> dict:
    span["started_at"] = utcnow()
    span["kind"] = "tool"
    try:
        completed = subprocess.run(argv, capture_output=True, text=True, timeout=timeout_s)
        span["ended_at"] = utcnow()
        span["status"] = "ok" if completed.returncode == 0 else "error"
        span.setdefault("attrs", {})["exit_code"] = completed.returncode
    except subprocess.TimeoutExpired:
        span["ended_at"] = utcnow()
        span["status"] = "timeout"
    except Exception as exc:
        span["ended_at"] = utcnow()
        span["status"] = "error"
        span.setdefault("attrs", {})["exception"] = type(exc).__name__
    return span

That except path is what turns a dropped connection on a free server into a closed error instead of a fake success. Use the wrapper clock for started_at and ended_at. Do not trust a duration the model narrates. Models round. Wrappers do not have to.

Keep raw logs. They are not the gate. A file diff is also not the gate. An empty diff plus an open apply_patch span is still a failed run, because the agent claimed work it never finished. The order is mechanical: close the spans, then read the diff, then read the model text.

A parent id on every tool is necessary and not sufficient. Ordering among stdout lines is a different bug. Completeness is coarser and cheaper. Did the span end, and did the root inherit that fact.

Limits, and who should skip this

The checker will not catch a tool that returned ok with the wrong bytes. It will not catch a model that never cited a tool it should have called. It will not reconstruct work if you never persisted spans. Cardinality still matters: do not stuff full file contents into attributes or the trace store becomes the bottleneck.

Skip the approach if the session has no tools, if the runtime already fails a parent when a child is open, or if the question you actually have is semantic eval, not structure. A completeness gate is a filter in front of those evals. It is not a substitute.

Vendor quotas, hardware SKUs, and model lists are omitted on purpose. Those figures move. A closed-span rule does not. If you already persist traces from a free model path, run the checker on a day's files before you change prompts. The prompt is rarely the first bug. An open apply_patch span usually is.