DEV Community: Sam Sun

Refuse the Next Attempt While the Trace File Is Shapeless

Sam Sun — Sat, 10 Oct 2026 19:17:43 +0000

A second agent attempt is a cost, not a diagnosis, until the first trace file has a closed shape. The summary line can say the run finished, and that sentence still does not prove a root span exists, that each tool call has a child, or that an error carries a type you can group later. Treat the trace file as a packing list. If a label is missing, do not order another shipment.

Debug loops fail quietly when the artifacts do not share an id. The model returns a paragraph, a tool log shows a command, and the working tree may change, yet none of those three is a span. The next call then starts a new story, and you cannot tell whether it retried the same tool, skipped it, or invented a third path. A shape check sits in front of that call: it does not decide whether a patch is correct, only whether the run is comparable at all.

Shape is the packing list, not the cargo. A clerk can confirm a crate has a label, a weight, and a seal without opening every item. Presence of a trace id, unique span ids, parent links, a non-negative duration, a tool name, and an input hash is that seal. Whether the hash matches the file you meant to edit is a later question, and mixing the two is how a team spends a model call on a missing field.

Keep the contract small enough to fail for one reason: one file, one trace id, exactly one root whose parent is empty, and every other span pointing at a parent that also appears in the file. Span ids do not repeat, and end_ns is at least start_ns, which only checks that single span's own clock values rather than ordering across machines. Spans named with a tool. prefix carry tool.name and input.sha256, while a status of error carries an exception event or an error.type attribute, and the root carries run.id. A missing field is an exit code, not a warning scrolled past in CI.

Exit code 2 should stop the shell the way a failed test stops a deploy. A warning still lets the next step run, and the next step is the model call you were trying to protect. Exit code 0 means the file can be joined to a later attempt. It does not mean the agent fixed the bug, and it does not mean a remote collector has the same bytes.

The checker below is a proposed local tool: it has not been timed, and it is not a benchmark. It reads a simplified JSON export rather than OTLP protobuf. If your tracer already writes OTLP JSON, map spanId, parentSpanId, and startTimeUnixNano in a short adapter, then keep the gate itself dull. One failure reason per run is the point.

#!/usr/bin/env python3
"""Proposed trace-shape gate. Unexecuted example; adapt names to your exporter."""
import hashlib, json, sys

def fail(msg: str) -> None:
    print(f"shape: {msg}", file=sys.stderr)
    raise SystemExit(2)

def main() -> None:
    path = sys.argv[1] if len(sys.argv) > 1 else "run.json"
    with open(path, encoding="utf-8") as fh:
        doc = json.load(fh)
    spans = doc.get("spans")
    if not isinstance(spans, list) or not spans:
        fail("spans missing")
    ids = [s.get("span_id") for s in spans]
    if any(not i for i in ids) or len(ids) != len(set(ids)):
        fail("span_id missing or duplicated")
    roots = [s for s in spans if not s.get("parent_span_id")]
    if len(roots) != 1:
        fail(f"expected 1 root, found {len(roots)}")
    if not doc.get("trace_id"):
        fail("trace_id missing")
    known = set(ids)
    for s in spans:
        try:
            if int(s["end_ns"]) < int(s["start_ns"]):
                fail(f"{s['span_id']} negative duration")
        except (KeyError, TypeError, ValueError):
            fail(f"{s.get('span_id')} bad timestamps")
        parent = s.get("parent_span_id")
        if parent and parent not in known:
            fail(f"{s['span_id']} parent not in file")
        attrs = s.get("attributes") or {}
        if s is roots[0] and not attrs.get("run.id"):
            fail("root missing run.id")
        if str(s.get("name", "")).startswith("tool."):
            if not attrs.get("tool.name") or not attrs.get("input.sha256"):
                fail(f"{s['span_id']} tool span missing name or input hash")
        if s.get("status") == "error":
            events = [e.get("name") for e in (s.get("events") or [])]
            if "exception" not in events and not attrs.get("error.type"):
                fail(f"{s['span_id']} error has no type")
    digest = hashlib.sha256(",".join(ids).encode()).hexdigest()[:12]
    print(f"shape_ok trace={doc['trace_id']} spans={len(spans)} id_hash={digest}")

if __name__ == "__main__":
    main()

Commit a fixture next to the script so the gate can be tested without an agent. The root needs run.id, and the tool span needs a hex input.sha256, status error, and error.type set to something like timeout. python3 trace_shape.py fixture.json should print one shape_ok line and exit 0; strip input.sha256 into a copy and expect exit 2. If the broken copy still exits 0, the gate is decorative and should not sit in front of any model call.

python3 trace_shape.py fixture.json; echo "pass:$?"
python3 - <<'PY'
import json
d = json.load(open("fixture.json", encoding="utf-8"))
d["spans"][1]["attributes"].pop("input.sha256", None)
json.dump(d, open("/tmp/broken.json", "w"), indent=2)
PY
python3 trace_shape.py /tmp/broken.json; echo "broken:$?"

The fixture itself can stay under thirty lines. Two spans are enough to prove the root rule and the tool rule. Add a third span later only when you have a second tool. Extra spans that exist only to make the file look serious will train you to ignore the checker.

{
  "trace_id": "4f0c9a11c0de4b0a9c11aa00bb11cc22",
  "spans": [
    {
      "span_id": "aa",
      "parent_span_id": null,
      "name": "agent.run",
      "start_ns": 1000,
      "end_ns": 9000,
      "status": "ok",
      "attributes": {"run.id": "run-001"},
      "events": []
    },
    {
      "span_id": "bb",
      "parent_span_id": "aa",
      "name": "tool.read_file",
      "start_ns": 1500,
      "end_ns": 4000,
      "status": "error",
      "attributes": {
        "tool.name": "read_file",
        "input.sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
        "error.type": "timeout"
      },
      "events": [{"name": "exception"}]
    }
  ]
}

Store the raw tool argument outside the span, keyed by that hash, if you still need the body for a local replay. The gate only checks that the hash field is present and does not recompute it, because the body is not in this file. Recomputation belongs in a different script so a redaction change is not mistaken for an agent change. The id_hash printed on success is only a fingerprint of span ids in file order, useful in a CI log, and it will change if you sort spans differently.

A behavioral diff is a second step, and it should wait until two files pass, at which point you compare names, status codes, and input.sha256 values. A name that exists only in the second file is a new tool call, and a hash that changes under the same tool name is a new argument. Comparing two files that failed the gate produces a neat table of absences, which feels like analysis and is not. This gate stops before that comparison on purpose.

Skip the approach when a collector already enforces the same schema and the agent cannot emit a local file, because a weaker second gate will drift from the first. Skip it for sampled production traffic too: a sampler that drops tool children will make the checker fail runs you chose to thin out, and the exit code will blame instrumentation for a sampling decision. Skip it if exit code 0 would be shown to anyone as proof the working tree is correct, since a well-shaped trace can describe a wrong edit. Skip it if you cannot keep even a hash of tool arguments, because that hash is still a join key and a join key needs an owner and a retention rule.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access matters here only as the budget a shapeless retry would waste, and the free server option matters only as a later export target after trace_shape.py has already passed on a local file, so collector setup and agent behavior are not debugged in the same hour. No model name, quota, hardware size, or retention period is stated, because none of those were verified for this draft; if that server is unreachable, keep the JSON, since a remote batch that never arrives neither creates nor erases a local pass. Once the file is green, that free access is a reasonable place for the next attempt, as long as the checked file stays the source of record.

The loop stays short on purpose: emit one JSON file, run the shape script, and repair instrumentation until the broken fixture exits 2 and the real file exits 0. Then, and only then, spend another model call. The summary line can wait. The packing list cannot.

Hold the Score Until the Collector Acks the Link Set

Sam Sun — Fri, 09 Oct 2026 15:38:14 +0000

A split-host agent run is not ready to score until the collector acknowledges every linked span you expected to export. Process exit only proves the local loop stopped. The trace you will compare is whatever a remote collector actually stored.

That gap shows up as soon as the model hop and the tool hop do not share a process. The model request is created on a service you do not parent. The tool span is created beside the working tree. A third host, the collector, knows only what arrived.

Scoring from the local log grades a draft. This note treats the gap as a hard gate. The code below is a proposed workflow, not an executed benchmark, and it reports no measured miss rate.

A parent pointer is the wrong join

Parent and child spans assume one process tree. You do not have that tree when the model call leaves the machine. Writing a local parent id onto a remote request invents a hierarchy the collector cannot check.

The relation you can defend is a link. This tool span is associated with that remote request id, and both ids must be present before a score is written. A link does not pretend the remote process was your child. It records a join key you can test.

The picture is a warehouse manifest beside a dock receipt. The manifest lists what left. The receipt lists what the dock accepted. A diff that reads only the manifest will count crates still on the truck.

A successful model status does not close the receipt. It says the model answered. It does not say the tool span, the link, and the model span landed in one trace. Wall-clock order does not close it either, because the two hosts do not share a clock you control.

Fingerprints that survive a model swap

A low-cost model swap helps a debug loop, and it ruins a naive diff. If the span name embeds a model id, two runs that called the same tools look unrelated. The label moved. The tool contract may not have moved at all.

Key the tool contract, not the vendor string. Canonicalize the tool name and the arguments, then hash them. Put the hash on the local span, and hash the result only when it is small and non-sensitive.

The comparison then asks whether the fingerprint sequence changed. It does not ask whether the model label changed. Stable hashes keep a model swap from looking like a regression.

Canonical form is part of the method. Sort keys, reject non-finite floats, and freeze separators. Otherwise a harmless reorder looks like a behavior change. Keep the canonicalizer in the same file as the gate so the expected set and the later diff cannot drift.

Raw arguments should not ride to a shared collector when they might hold secrets. The fingerprint can travel. The body stays in a local file you open only after a hash differs. That split is a limit of this loop, not a full redaction program.

The gate, as proposed code

The script has not been run for this article. It encodes one rule. Do not emit a ready score unless the ack covers every expected id, including the remote request id. Save it as ack_gate.py if you want the shell step below to import it. The type hints assume Python 3.9 or newer.

#!/usr/bin/env python3
"""Proposed gate. Unexecuted example: hold the score until the ack covers the set."""

import hashlib
import json
import sys
from pathlib import Path

def canonical(payload: dict) -> bytes:
    return json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
        allow_nan=False,
    ).encode()

def fingerprint(payload: dict) -> str:
    return hashlib.sha256(canonical(payload)).hexdigest()

def build_expected(run_id: str, remote_request_id: str, calls: list) -> dict:
    spans = []
    for index, call in enumerate(calls):
        spans.append({
            "span_id": f"{run_id}-tool-{index:03d}",
            "run_id": run_id,
            "link": {"remote_request_id": remote_request_id},
            "tool_fingerprint": fingerprint({"name": call["name"], "args": call["args"]}),
            "result_fingerprint": fingerprint({"result": call.get("result")}),
        })
    return {
        "run_id": run_id,
        "remote_request_id": remote_request_id,
        "spans": spans,
    }

def score_allowed(expected: dict, ack: dict) -> bool:
    wanted = {span["span_id"] for span in expected["spans"]}
    wanted.add(expected["remote_request_id"])
    return wanted <= set(ack.get("accepted_ids", []))

def main() -> int:
    expected = json.loads(Path(sys.argv[1]).read_text())
    ack = json.loads(Path(sys.argv[2]).read_text())
    if not score_allowed(expected, ack):
        wanted = {span["span_id"] for span in expected["spans"]}
        wanted.add(expected["remote_request_id"])
        missing = sorted(wanted - set(ack.get("accepted_ids", [])))
        print(json.dumps({"status": "hold", "missing": missing}))
        return 2
    sequence = [span["tool_fingerprint"] for span in expected["spans"]]
    print(json.dumps({
        "status": "ready",
        "run_id": expected["run_id"],
        "sequence": sequence,
    }))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Build the expected file from the calls you actually made, before you export. The numbers in the fixture are placeholders, not measurements.

python3 - <<'PY'
import json
from ack_gate import build_expected

calls = [
    {"name": "read_file", "args": {"path": "src/app.py"}, "result": {"bytes": 1204}},
    {"name": "run_tests", "args": {"target": "tests/test_app.py"}, "result": {"failed": 0}},
]
doc = build_expected("run-1842", "req-9f3a", calls)
open("expected.json", "w").write(json.dumps(doc, indent=2) + "\n")
PY
python3 ack_gate.py expected.json ack.json

Exit 2 means hold, and the JSON lists missing ids. Exit 0 means the fingerprint sequence may be compared with another ready run. A 202 from an async ingest URL is a queue ticket, not an ack file.

Persist the ack only from a body that names accepted ids. Include the remote request id in the same expected set as the tool spans. A tool-only ack still hides the model side, so the gate holds.

Span events are not a substitute for that body. An event can be sampled away while the span id still arrives, or the reverse. The score waits on ids the collector lists, not on events you hoped to attach.

Walk one failed export before you trust the script. Suppose the model span arrives and the second tool span does not, because the batch flushed early. The missing list contains run-1842-tool-001, and the sequence is withheld.

Retry the export with the same span ids. Replace ack.json with the new body, then run the gate again. Do not average the two acks, and do not mark the run partial-ready. Partial-ready is how a later diff blames the model for a span that never landed.

Idempotency belongs on the export, not on a fresh id per attempt. Reuse run-1842-tool-001 when you resend. A new id on each retry makes the expected set chase a moving target, and the ack can never catch it.

Where the free path fits

You can practice the split without standing up a private collector. MonkeyCode is the open-source project used here as the remote side: free model access for the model hop, and a free server option for the collector hop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both are availability claims, not a capacity plan. This article does not name models, token quotas, machine sizes, retention, or uptime. Those figures move, and a stale quota in a debug script is worse than no quota at all.

Read the current project documentation before you bind the loop to a limit. Do not paste an old token budget into the gate. If the docs and the runtime disagree, trust the runtime error and update the note.

Point the model hop at the free model access. Point the exporter at the free server, and send ids plus fingerprints rather than raw tool bodies. Keep expected.json beside the run directory so review does not depend on a dashboard session.

If the server schema cannot list accepted ids, do not fill them in locally. Extend the export until the ack is explicit, or leave the run on hold. A locally invented ack is just the manifest copied onto receipt paper.

The review step stays on the machine that ran the tools. When a ready sequence changes, open the local tool log for that run id and note the first hash that moved. The shared server is an index. It does not declare the agent fixed.

Who should not use this

The gate catches a missing export. It does not catch a wrong result that was exported in full. You still need an oracle on the fingerprint or on the local outcome.

It assumes the ack is specific and repeatable. If the server only says ok, the script holds forever, unless someone stubs accepted_ids with the expected set and voids the check. Do not stub it.

Do not use a free server option as audit storage, incident evidence, or a system of record. A convenience collector can drop or reset data. Do not send tool arguments that contain credentials, customer data, or private source.

Skip the gate if the model and the tools already share one process and one tracer. The extra files are ceremony you will ignore. Skip it if you cannot change the exporter to name accepted ids.

A gate without an honest ack is a new way to feel finished. Waiting costs whatever the batch interval costs. That wait is the method. A faster score on a partial trace is a confident miss.

Close on three files

Keep the expected set, the collector ack, and the fingerprint sequence. Diff sequences only across runs whose acks covered their sets. If the sequence is stable and the local oracle passed, record the run as unchanged.

If the sequence moved, inspect the local body for the first differing hash and stop. Do not keep walking the trace until you find a story that fits. The first hash delta is the debug pointer.

That is smaller than a claim that the agent is fixed. It is a claim you can recheck after the model label changes and the tool contract does not. A later swap should move the vendor string and leave the sequence alone, or the contract really changed.

If run notes already sit in the working tree, write the expected set there and export only ids and fingerprints. Recheck the current free model access and free server terms in the project docs. Let the ack file, not a status page, decide whether the run is scorable.

Keep Tool Bodies Local When Spans Leave the Machine

Sam Sun — Thu, 08 Oct 2026 11:00:38 +0000

A remote agent trace is a reliable diff only after every tool call has been closed on the machine that ran it, and only after arguments and results have been replaced by stable hashes. Exported span text is a label. The label can match while the body was truncated, reordered by a collector, or dropped to fit an attribute budget you do not control.

That constraint shows up as soon as a demo leaves a laptop. Public developer discussion this week has been heavy with agent demos, hackathon calls, and portfolios assembled quickly with model help. Those pieces can be fine as demos. They are a poor debug log. A green summary after two runs does not tell you whether the same tool was invoked, whether the arguments matched, or whether a collector kept the fields you meant to compare.

Treat the export path like a courier that photographs the crate label and may discard the crate. If the photograph is all you archive, the next regression will argue about photography. The loop below keeps the crate in a local store and ships the photograph on purpose. A warehouse of labels can still tell you that a call named apply_patch left with status ok. It cannot tell you which bytes were in the patch if the photo cropped the middle of the label. Teams then "fix" the diff by comparing cropped strings, and the crop quietly becomes the spec.

The loop has four beats, and it stays on one machine until the last beat. First, wrap the tool boundary so entry and exit are recorded in the same process that executed the call. Second, redact, canonicalize, hash, and write the canonical body to a content-addressed directory. Third, refuse export while any call lacks a terminal status or any hash lacks a local file. Fourth, diff two runs by the multiset of tool name, status, argument hash, and result hash. A summary string is not an input to that diff. Length rides along so a 40 KB patch and a 400 byte patch cannot masquerade as the same event before anyone opens the body files.

Canonicalization is the part people skip, and it is the part that makes the hash mean something. JSON objects do not have a stable key order. A model that emits one key order and a wrapper that emits another are the same call if you sort keys and fix separators before hashing. They are a false regression if you hash the raw string a logger happened to print. Timestamps, random request ids, and absolute temp paths do not belong inside the hashed body. Strip them into attributes that you do not use for equality, or the gate will fail closed on noise.

Redaction sits before hashing, not after. If an argument contains a token, a customer record, or a signed URL, replace that field with a typed placeholder and hash the placeholder form. Hashing a secret and exporting only the hash is better than exporting the secret, but the local store would still hold the secret if you write the raw value. Write the redacted canonical form, and keep the real secret in a system that already has an access story. This note is not a redaction product. It only fixes the order: redact, canonicalize, hash, store, then consider export.

The listing that follows is a proposal. It was not executed against a live collector for this draft, and it reports no timings, quotas, or pass rates. Non-JSON payloads need a separate byte canonicalization. This version assumes JSON-compatible values.

#!/usr/bin/env python3
"""Local hash gate before an agent trace export. Proposal, not a measured run."""
import hashlib, json, pathlib
from collections import Counter

def canonicalize(value):
    return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

def payload_hash(value):
    body = canonicalize(value).encode("utf-8")
    return hashlib.sha256(body).hexdigest(), len(body)

def tool_span(name, args, result, status):
    args_hash, args_len = payload_hash(args)
    result_hash, result_len = payload_hash(result)
    return {
        "name": name,
        "status": status,
        "args_sha256": args_hash,
        "args_len": args_len,
        "result_sha256": result_hash,
        "result_len": result_len,
    }

def write_bodies(store, span, args, result):
    root = pathlib.Path(store)
    root.mkdir(parents=True, exist_ok=True)
    for digest, body in ((span["args_sha256"], args), (span["result_sha256"], result)):
        path = root / digest
        text = canonicalize(body)
        if not path.exists():
            path.write_text(text, encoding="utf-8")
        elif path.read_text(encoding="utf-8") != text:
            raise RuntimeError(f"unstable canonical form: {digest}")

def export_ready(spans, store):
    problems, root = [], pathlib.Path(store)
    for span in spans:
        if span["status"] not in {"ok", "error"}:
            problems.append(f"open:{span['name']}")
        for key in ("args_sha256", "result_sha256"):
            if not (root / span[key]).is_file():
                problems.append(f"missing:{span[key]}")
    return problems

def manifest(spans):
    return [(s["name"], s["status"], s["args_sha256"], s["result_sha256"]) for s in spans]

def diff_manifests(left, right):
    cl, cr = Counter(map(tuple, left)), Counter(map(tuple, right))
    return {
        "only_left": sorted((cl - cr).elements()),
        "only_right": sorted((cr - cl).elements()),
    }

A command-shaped check, still illustrative, is short enough to alias. Record two fixtures, refuse the push when the gate prints any problem, and diff manifests only after both gates are empty.

python trace_gate.py run --fixture fixtures/rename_symbol.json --store runs/a/bodies --out runs/a/spans.json
python trace_gate.py gate --spans runs/a/spans.json --store runs/a/bodies
python trace_gate.py diff --left runs/a/spans.json --right runs/b/spans.json

The run subcommand is intentionally not fully spelled out. Wire it to the agent runner so the same function both invokes the tool and calls tool_span plus write_bodies. A logger that scrapes stdout after the fact will hash a formatted line, not the value the tool received, and the gate will bless a fiction. Fixture choice matters more than collector choice. A fixture that calls the live network will hash a moving world. Pin the tool doubles. A rename fixture that reads three files from a temp repository, applies one patch, and writes a unified diff is enough. On the next run, a model that reflows the patch shows up as only_right, which is the signal you wanted, not a collector glitch.

What leaves the machine is the span record: tool name, status, two hashes, two lengths. What stays is the body directory. That split is the decision rule. If a body is small, already redacted, and you have a written reason to debug it remotely, you may attach it as an attribute. If you cannot state the collector's current attribute budget, do not attach it. A budget you guess will be wrong on the day the cap tightens, and a truncated JSON attribute will not match the local body. Identical runs then look flaky, and the flake is in the courier.

MonkeyCode's free model access and free server option, as described for this workflow by the operator, fit on the two sides of that split. Free model access is a way to repeat a pinned fixture often enough that a hash diff has more than one sample. A free server is a place the reduced spans can land when you do not want to stand up a collector just to see the manifest. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This draft does not restate model names, token quotas, hardware, retention, or duration. Those figures change, and an article is the wrong cache for them. Read the current project documentation before you depend on a limit, and treat anything not written there as unknown.

The free server does not relax the gate. A hosted collector is a stronger reason to hash locally, because you cannot assume which attributes survive the hop, and you should not put raw tool arguments on a machine you do not operate just to make a diff easier. Replay still happens from the local body store. The remote copy is an index. After the server accepts a batch, fetch the stored spans once and check that each args_sha256 you sent is still present and unchanged. If the read-back drops a hash, stop using that path for regression. A missing hash is not a partial success. It means the courier photographed a label and then smudged it. Ordering across hosts is a separate check. Do not fold clock skew into the hash, or every machine becomes a unique run.

Hash equality will not save a semantic bug. Two patches can differ by a comment and produce different result hashes while both fix the build. Two identical hashes can still hide a bad patch if the fixture never executed the failing path. The gate answers a narrow question: did the tool boundary see the same recorded inputs and emit the same recorded outputs as the run you kept? It does not answer whether the outputs were wise. Generation text that never passed through a tool needs its own fingerprint, stored beside the manifest, not stuffed into the tool hash. When the manifest diff is empty and the read-back matches, you still have not proved the agent is fixed. You have proved the tool boundary was stable across the two runs you kept. Hold the loop to that smaller bar.

Do not use this loop where the arguments are data you are not allowed to write to local disk either. A content-addressed store is still a store. Do not use it as a compliance archive. Nothing here states a retention period, an access policy, or a deletion story for a free server. Do not use it to judge sampler-only bugs. If the only difference between runs is a temperature draw inside the model, the manifests will flap and the flap will look like a tool regression. Fix the fixture, or stop diffing those fields. And do not bypass the gate because a demo deadline wants spans on a dashboard tonight. That bypass is how truncated attributes become the source of truth.

The useful close is boring, which is what you want from a debug loop. Keep the bodies next to the run, export the hashes, and let the remote index earn trust only after a read-back. If that index is a free server, confirm today's limits in the docs, then point the exporter at the reduced record rather than at the crate.

Block the Retry When the Tool Span Already Failed

Sam Sun — Thu, 08 Oct 2026 08:44:38 +0000

A repeated tool span with the same argument hash and a prior failed exit is not new evidence. It is the same failure, recorded twice. When the next step would spend a free model call on a free server, that repeat is the wrong unit of spend. The chat summary cannot make the second attempt distinct.

Agent debug loops still treat the final message as the thing to judge. That message is a compressed claim. The checkable unit is smaller: the tool name, a fingerprint of its arguments, the tool version, the process exit, and the diff left in the tree. If those fields already sit in a closed local row, another remote run cannot change the conclusion. It can only add latency and another unread log.

A returns desk that restocks a cracked mug because the receipt says completed has made the same mistake. The mug is the artifact. The receipt is the summary. A trace has to follow the mug. Public discussion this week leans toward portfolios, meetups, and profile showcases. Those are fine topics. They do not tell a reviewer whether a tool call already failed.

Close the span, then ask

The module below is an unexecuted proposal. It is not a measured tracer, and it is not a claim about any vendor's export format. It keeps a JSON ledger on the laptop. Raw prompts and raw arguments stay out of the file. A short SHA-256 prefix is enough to recognize a copy-paste retry inside one debug session.

#!/usr/bin/env python3
"""Unexecuted proposal: veto a retry from a local tool-span ledger."""
import hashlib
import json
from pathlib import Path

LEDGER = Path(".trace/tool-span-ledger.json")

def fingerprint(tool: str, arg_text: str, tool_version: str) -> str:
    payload = f"{tool}\n{tool_version}\n{arg_text}".encode()
    return hashlib.sha256(payload).hexdigest()[:16]

def load() -> dict:
    if not LEDGER.exists():
        return {"spans": []}
    return json.loads(LEDGER.read_text())

def record(tool, arg_text, tool_version, exit_code, diff_stat, trace_id, span_id):
    entry = {
        "fp": fingerprint(tool, arg_text, tool_version),
        "tool": tool,
        "tool_version": tool_version,
        "exit_code": exit_code,
        "diff_stat": diff_stat,
        "trace_id": trace_id,
        "span_id": span_id,
    }
    data = load()
    data["spans"].append(entry)
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    LEDGER.write_text(json.dumps(data, indent=2) + "\n")
    return entry

def veto_retry(tool: str, arg_text: str, tool_version: str) -> str:
    fp = fingerprint(tool, arg_text, tool_version)
    prior = [s for s in load()["spans"] if s["fp"] == fp]
    if not prior:
        return "allow: no prior span"
    last = prior[-1]
    if last["exit_code"] != 0:
        return f"block: span {last['span_id']} already exited {last['exit_code']}"
    if last["diff_stat"] in ("", "0 files changed"):
        return f"block: span {last['span_id']} closed with an empty diff"
    return "allow: last span exited 0 with a non-empty diff"

veto_retry is the decision, not a suggestion buried in a paragraph. allow means a new remote run can add information. block means the next model call would restate a span the ledger already closed. An empty diff counts as a failed advance even when the process exit is zero. A green exit that changes no file does not move a code fix forward.

A stderr line joins that row only when it carries the same trace_id and span_id. Otherwise it remains a side channel. Unjoined logs can explain a failure after the veto. They cannot overturn the veto by themselves. A working-tree stat joins the same way: git diff --stat is stored as text on the span, not inferred later from a summary that forgot the path.

One failed formatter, then a stop

Take a formatter invoked as ruff check src at tool version 0.6.9. The child exits 1. git diff --stat prints nothing, because the tool only reported diagnostics. That pair is the evidence. A later agent turn that proposes the same command, the same version, and the same arguments is the same span in a new coat of prose.

mkdir -p .trace
python3 - <<'PY'
from span_ledger import record, veto_retry
record("ruff", "check src", "0.6.9", 1, "", "7f3a", "c21")
print(veto_retry("ruff", "check src", "0.6.9"))
print(veto_retry("ruff", "check src --fix", "0.6.9"))
PY

The expected lines are block: span c21 already exited 1 and then allow: no prior span. The second call changed the argument text, so the fingerprint changed. That is the only kind of retry this loop treats as new work. A fresh trace id is not new work. A fresh span id is not new work. Identity lives in the tool, the version, and the argument text.

The shell fragment assumes the proposal lives in span_ledger.py on PYTHONPATH. It has not been run as a published benchmark. Wiring argparse is optional. The order is not optional: record the closed span, run the veto, and only then decide whether a remote execution is eligible. Reversing that order is how a free run gets spent on a transcript the laptop already settled.

If the tree was dirty for reasons the tool did not own, add a tree hash to the fingerprint inputs before trusting allow. A dirty file that the argument text never mentions will otherwise hide inside a repeated hash. The ledger is only as honest as the fields it is given.

Spend the free run only on allow

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode belongs in this loop only as a destination for an eligible retry, not as the source of the verdict. The operator states that free model access and a free server option exist. This draft does not state a token quota, a hardware shape, a retention window, a model name, or a license, because those figures were not checked against a primary source on 2026-10-08. A dashboard number stays unread until it is checked on the day of the run.

The sequence is local first. Close the tool span. Write the ledger row. Run veto_retry. Point only an allow result at the free server. When the remote summary returns, join it to the new span id before anyone calls the bug fixed. If the export lacks an exit code, the local process status wins. A missing export is an incomplete span, not a success.

That order still holds if the product name is removed. Any model route with a limited run budget has the same constraint. The free option only makes the cost visible: the next person in the queue waits, or the same person spends the run that should have tested a changed fingerprint. If free model access and a free server are already on the bench, send them the allow path and leave the block path on disk.

What a 16-hex prefix can say

No timing result is claimed, and no model comparison is implied. Sixteen hexadecimal characters are 64 bits of a SHA-256 digest. That width is enough to catch copy-paste retries in a single laptop session. It is too weak to serve as a global identifier across teams. Collisions are not the decision anyway. The stored tool name and version sit beside the hash, and the veto reads all three.

The rule does not measure model quality. It measures whether the tool attempt changed. Two drafts can differ in prose and still call the same tool with the same arguments. The ledger blocks that second call on purpose. If the failure was a flake, a full disk, or a clock jump, the operator has to change an input the fingerprint can see: the arguments, the tool version, or a recorded tree hash. Until one of those changes, another sample is not a new experiment.

Secrets are the hard limit. Argument text is hashed, not stored, but the digest is still derived from a secret if a token was passed on the command line. Do not feed raw credentials into fingerprint. Redact first, hash the redacted form, and keep the redaction rule stable. A shifting redaction makes every retry look new, which quietly disables the veto.

Who should skip the ledger

Skip it when a trace backend already enforces idempotency keys on tool calls and the team pages from those keys. A second local file will drift, and drift is a second source of false certainty. Skip it for live incident response, where repeating a call may be the mitigation rather than a wasted debug retry. A veto built for code-fix loops will block a probe that was supposed to differ in network state.

Skip it when the tool output depends on time, an unread file, or a service the fingerprint cannot see. In that case the veto refuses a run that might have changed. That refusal is safer than a false fix, and it is still the wrong tool for an exploratory probe. Also skip it when the goal is a public demo that needs visible motion. This loop is a refusal mechanism. The session will look quieter. Quieter is the result.

The artifact to keep is the blocked line: span id, exit code, and the reason another free run was not started. A summary that never names that span is not a close. It is the receipt, waiting for someone to look at the mug.

A Parent Success Span Can Outlive a Dropped Tool Child

Sam Sun — Thu, 24 Sep 2026 22:32:38 +0000

A parent span marked OK does not close an agent debug loop. Head-based sampling keeps that parent and can drop the failed tool child, so a dashboard reports a healthy run while the working tree is unchanged. Close the loop only after an error-biased sampler, a content hash of the tool arguments, and a local diff record all agree on the same tool_call_id.

The failure looks like a hospital that files the discharge summary and shreds the abnormal lab. The visit has an end time. The evidence that the treatment missed is gone. Agent traces fail the same way when the sampler decides at the start of the trace, before any tool has exited.

What head sampling actually keeps

Head sampling chooses a keep-or-drop bit when the trace starts. That bit is cheap, and it is also blind. A later tool can return a non-zero exit, an empty patch, or no end event at all, and the child span is already doomed if the head decision was drop.

Tail sampling waits until the trace has a status. It costs more, because spans must sit in a buffer until the decision. It is also the common policy that can promise to keep every child whose status is ERROR, UNSET, or missing an end. The parent may still be sampled down. The failure should not be.

None of this is a claim about a vendor default. It is a property of the two policies. If you do not know which policy sits in front of a free remote runner, measure it. Do not infer it from a green parent.

A join the dashboard will not do for you

Picture one coding-agent turn. The model span lists two tool calls, edit_file and run_tests. The remote runner executes them. Your laptop holds the git diff. Three records exist, and none of them is the truth alone.

The model span can claim both calls were issued. The runner can record one exit and lose the other to a full buffer. The diff can show a one-line change that matches neither call if a human edited the tree between runs. A reusable loop treats the run as open until each issued tool_call_id has a surviving child and a diff hash, or an explicit error status that explains why no diff exists.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project with free model access and a free server option. Those two availability claims are why this loop is practical to rehearse: a model route and a remote runner can share one trace id without first buying a trace-vendor seat. Current quotas, model names, retention, and hardware are not stated here, because they were not verified against a primary source for this draft. Read the project page before you depend on a limit. The checker below does not require MonkeyCode. Any runner that emits the fields will do.

Fields worth keeping when the payload is not

Free remote hops often cap attribute size. Stuffing a whole file into a span attribute is how the tail of a tool call disappears, after which the truncation looks like a model mistake. Store a bounded preview and a hash. Keep the raw payload, if you must keep it at all, in a local artifact directory keyed by trace id, and redact secrets before either copy is written.

The envelope is small on purpose. It is an unexecuted reference snippet, not a measured production client.

import hashlib

PREVIEW_BYTES = 240

def arg_envelope(payload: bytes) -> dict:
    digest = hashlib.sha256(payload).hexdigest()
    return {
        "arg_sha256": digest,
        "arg_bytes": len(payload),
        "arg_preview": payload[:PREVIEW_BYTES].decode("utf-8", errors="replace"),
    }

A child span then needs trace_id, span_id, parent_span_id, tool_call_id, status_code, arg_sha256, and, after the runner finishes, exit_code plus diff_sha256. The preview is a hint for humans. The hash is what the joiner trusts. If the preview is cut mid-character, the hash still identifies the original bytes, so a later fetch from the local artifact store can restore the full argument without trusting the truncated attribute.

Sampler policy as code, not as a slogan

The following module is a proposal. It has not been executed against a live server for this article, and it is not a description of any product sampler. It shows the decision you want in front of a runner whose policy you have not measured.

from dataclasses import dataclass

@dataclass(frozen=True)
class Span:
    span_id: str
    parent_span_id: str | None
    tool_call_id: str | None
    status_code: str  # OK, ERROR, UNSET
    ended: bool

def keep_span(span: Span, head_keep: bool) -> bool:
    if span.tool_call_id and (span.status_code != "OK" or not span.ended):
        return True
    return head_keep

OK tool children may still be dropped when head_keep is false. That is acceptable only if a separate diff ledger already recorded diff_sha256 for that tool_call_id. If the ledger is empty and the child is gone, you no longer have a debug loop. You have a story.

Think of the head bit as a coin flipped in the lobby. The coin does not know which lab result will come back abnormal. Error-biased retention is the clerk who refuses to shred a folder once a red stamp appears, even if the lobby coin said drop. The analogy stops at the policy. You still have to implement the stamp, which here is status_code plus ended.

The joiner

Export three JSONL files. issued.jsonl comes from the model span's tool-call list. children.jsonl comes from whatever the remote runner actually retained. diffs.jsonl is local, written by your wrapper after git diff. The joiner below is unexecuted reference code. Adapt the field names if your exporter uses a different key, but do not join on wall-clock timestamps.

import json, sys

def load(path):
    with open(path, encoding="utf-8") as fh:
        return [json.loads(line) for line in fh if line.strip()]

def open_calls(issued, children, diffs):
    by_call = {row["tool_call_id"]: row for row in children}
    diff_ids = {row["tool_call_id"] for row in diffs if row.get("diff_sha256")}
    still_open = []
    for row in issued:
        call_id = row["tool_call_id"]
        child = by_call.get(call_id)
        if child is None:
            still_open.append((call_id, "dropped_child"))
        elif not child.get("ended") or child.get("status_code") == "UNSET":
            still_open.append((call_id, "unclosed_child"))
        elif child.get("status_code") == "ERROR":
            still_open.append((call_id, "tool_error"))
        elif call_id not in diff_ids:
            still_open.append((call_id, "missing_diff"))
    return still_open

if __name__ == "__main__":
    issued, children, diffs = map(load, sys.argv[1:4])
    pending = open_calls(issued, children, diffs)
    for call_id, reason in pending:
        print(f"{reason}\t{call_id}")
    sys.exit(1 if pending else 0)

A wrapper can build the diff ledger without a trace vendor. The hash is of the patch text, not of the whole repository, so two runs that touch the same line in different ways do not collide. Generate the trace id before the first remote hop, and pass that same id into the model route, the runner, and this local file.

TRACE=4f0c0c2a9b1e4d77a0c1
export TRACE TOOL_CALL_ID=call_edit_01
git diff --binary > "/tmp/${TRACE}.patch"
python - <<'PY'
import hashlib, json, os
trace = os.environ["TRACE"]
blob = open(f"/tmp/{trace}.patch", "rb").read()
row = {
  "trace_id": trace,
  "tool_call_id": os.environ["TOOL_CALL_ID"],
  "diff_sha256": hashlib.sha256(blob).hexdigest(),
  "diff_bytes": len(blob),
}
open("diffs.jsonl", "a", encoding="utf-8").write(json.dumps(row) + "\n")
PY
python join_open_calls.py issued.jsonl children.jsonl diffs.jsonl

Exit status is the contract. No lines printed means every issued call joined a closed child and a diff hash. Any printed line means the run stays open, even if the parent status is OK. A silent report is the only pass condition. One green parent is not.

Count the reasons across repeated runs before you tune a prompt. If dropped_child dominates, the sampler or the exporter is the defect. If missing_diff dominates, the runner is reporting success without a tree change. If tool_error dominates, the model route is not the first place to look. Those ratios are the data. A single green trace is an anecdote.

You can rehearse that count on a free model route and a free server, then move the same three files to any other runner. The route supplies generations. The server supplies tool exits. The laptop supplies the diff hash. Remove any one of the three and the parent status becomes a rumor again.

What this will not prove

The join proves recording integrity, not that the patch is correct. A wrong but present diff still hashes. Semantic review stays outside this loop. Treat a matching hash as a gate that lets a human or a later test look at the patch, not as a verdict on the patch.

Tail retention of every error can fill a small free disk when a retry storm emits thousands of failed tool children. Cap the local artifact store by trace id and by age, and drop previews before you drop hashes. A hash without a preview is still joinable. A preview without a hash is not.

Do not put secrets, tokens, or customer payloads into span attributes or into the preview field. The envelope function above does not redact. Run a redaction step before arg_envelope, or do not use this pattern on those traces. A free server is still a remote hop. What you attach to a span can leave the laptop.

Skip the approach if you need legal hold of full prompts, multi-tenant access control, or a guaranteed retention window. A free server option is a place to rehearse the loop, not a promise about durability. Also skip it if your runner cannot emit tool_call_id on both the model side and the tool side. Without that shared key, you are back to joining on timestamps, and timestamps across machines do not order causality.

If a free model route and a free server are already available to you, point one failing run at this joiner before you trust the parent status again. Confirm the current access terms on the project page first. The parent can outlive the child. The child is the run.

Unpromoted Tool Logs Cannot Close a Debug Loop

Sam Sun — Wed, 23 Sep 2026 17:37:30 +0000

An agent debug session that starts in stdout is already looking at a side channel. The unit that can be compared, retried, and blamed is a closed span with events. A tool log line that never promotes into that span is not extra context. It is a failed observation.

Most coding agents still print what they did. The printer is convenient. It is also a shuffle. Two tools that run in parallel write overlapping chunks, a retry writes a second success after a timeout, and the model summary then narrates a single story. None of those three artifacts share a join key unless you force one into the child process that actually ran the tool.

Think of the log file as a loading dock and the span tree as inventory. Boxes that never get a pallet tag do not belong on the warehouse floor. You do not keep them nearby in case they help. You fail the receiving check and stop talking about root cause.

Promotion is a copy, not a reading of the chat

Promotion is mechanical. Each tool start and tool exit must carry a stable tool_call_id. That id is the pallet tag. The span with the same id is the pallet. The log line becomes a span event whose time comes from the tracer, not from the laptop clock that printed the line.

If the id is missing, the line stays on the dock. The run is incomplete. That is the entire method. It does not score the model and it does not parse English. After ingest, the only question is whether any tool-shaped log line remains.

Agents often shell out. The child inherits environment, or it does not. If the id lives only in the parent's memory, the child's stdout is already unjoinable. Put the id where the child can echo it, then refuse pretty paragraphs as the record of record.

export AGENT_RUN_ID="run_7f3c"
export TOOL_CALL_ID="call_19"
python tools/grep_repo.py --pattern "TODO" --json

The tool should write one NDJSON object to a side file or to stderr, not a prose status line:

{"event":"tool.exit","run_id":"run_7f3c","tool_call_id":"call_19","status":"ok","bytes_out":412}

The tracer records the same ids on the span. Progress bars, token streams, and colored diffs are optional. They are also discardable. Do not use wall-clock order as a substitute key. Two parallel grep calls can finish inside the same millisecond window on a loaded machine, and clocks jump. Ids do not.

Completeness is a test, not a dashboard

Dashboards forgive holes. Tests do not. The checker below reads a span dump and a raw tool log, promotes what it can join, and fails the process if anything tool-shaped remains. Treat it as a proposed gate. It is not a production tracer. It assumes NDJSON spans and one regex for tool lines, and both must change when your agent format changes.

#!/usr/bin/env python3
"""Promote tool log lines into span events. Fail on leftovers.

Proposed checker. Point SPAN_PATH and LOG_PATH at real dumps
before treating an exit code as evidence about an agent.
"""
from __future__ import annotations

import json
import os
import re
import sys
from typing import Any

TOOL_LINE = re.compile(
    r"tool\.(start|exit)\s+id=(?P<id>[A-Za-z0-9_\-]+)"
    r"(?:\s+status=(?P<status>\w+))?"
)

def load_ndjson(path: str) -> list[dict[str, Any]]:
    rows = []
    with open(path, encoding="utf-8") as fh:
        for line_no, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                rows.append(json.loads(line))
            except json.JSONDecodeError as exc:
                raise SystemExit(f"{path}:{line_no}: {exc}") from exc
    return rows

def index_spans(spans: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
    by_tool = {}
    for span in spans:
        attrs = span.get("attributes") or {}
        tool_id = attrs.get("tool_call_id")
        if not tool_id:
            continue
        if tool_id in by_tool:
            raise SystemExit(f"duplicate tool_call_id on spans: {tool_id}")
        span.setdefault("events", [])
        by_tool[tool_id] = span
    return by_tool

def promote(spans_by_tool: dict[str, dict[str, Any]], log_path: str) -> list[str]:
    leftovers = []
    with open(log_path, encoding="utf-8") as fh:
        for line_no, raw in enumerate(fh, 1):
            raw = raw.rstrip("\n")
            m = TOOL_LINE.search(raw)
            if not m:
                continue
            tool_id = m.group("id")
            span = spans_by_tool.get(tool_id)
            if span is None:
                leftovers.append(f"{log_path}:{line_no}: no span for {tool_id}")
                continue
            span["events"].append(
                {
                    "name": f"tool.{m.group(1)}",
                    "attributes": {
                        "tool_call_id": tool_id,
                        "status": m.group("status"),
                        "log_line": line_no,
                    },
                }
            )
    return leftovers

def closed_without_exit(spans_by_tool: dict[str, dict[str, Any]]) -> list[str]:
    missing = []
    for tool_id, span in spans_by_tool.items():
        names = {ev.get("name") for ev in span.get("events") or []}
        if "tool.exit" not in names:
            missing.append(
                f"span {span.get('span_id')} tool_call_id={tool_id} has no tool.exit"
            )
    return missing

def main() -> int:
    span_path = os.environ.get("SPAN_PATH", "spans.jsonl")
    log_path = os.environ.get("LOG_PATH", "tools.log")
    spans = load_ndjson(span_path)
    by_tool = index_spans(spans)
    leftovers = promote(by_tool, log_path)
    missing_exits = closed_without_exit(by_tool)
    report = {
        "spans": len(spans),
        "tool_spans": len(by_tool),
        "promoted_events": sum(len(s.get("events") or []) for s in by_tool.values()),
        "leftover_tool_lines": leftovers,
        "spans_missing_exit": missing_exits,
    }
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    if leftovers or missing_exits:
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A fixture pair makes the failure mode obvious. Save this as dump/spans.jsonl:

{"span_id":"s1","parent_span_id":null,"start_unix_nano":1000,"end_unix_nano":4000,"attributes":{"tool_call_id":"call_19"},"events":[]}
{"span_id":"s2","parent_span_id":"s0","start_unix_nano":1500,"end_unix_nano":3900,"attributes":{"tool_call_id":"call_20"},"events":[]}

And this as dump/tools.log:

tool.start id=call_19
tool.exit id=call_19 status=ok
tool.start id=call_20
tool.exit id=call_21 status=ok

call_21 has no pallet. The gate should refuse to open.

export SPAN_PATH=./dump/spans.jsonl
export LOG_PATH=./dump/tools.log
python promote_tool_logs.py
echo $?

Exit code 2 means the debug loop must not start. There is nothing to interpret yet. The model may have printed a confident summary. The summary is downstream of missing evidence, the way a shipping label is downstream of an empty bay.

Parallel tools still have no timeline

Promotion does not give you order. Two events on different spans are concurrent unless a parent link or an explicit span link says otherwise. Concatenating tools.log after the fact is a lexical sort of a race. A second helper should make that refusal explicit. It does not invent happens-before edges.

def concurrent_pairs(spans: list[dict[str, Any]]) -> list[tuple[str, str]]:
    """Return tool_call_id pairs whose intervals overlap.

    Overlap means neither span is parent of the other and each
    starts before the other ends. Proposed helper, not a scheduler.
    """
    tools = []
    for span in spans:
        attrs = span.get("attributes") or {}
        tid = attrs.get("tool_call_id")
        if not tid:
            continue
        tools.append(
            (
                tid,
                int(span["start_unix_nano"]),
                int(span["end_unix_nano"]),
                span.get("parent_span_id"),
                span.get("span_id"),
            )
        )
    pairs = []
    for i, a in enumerate(tools):
        for b in tools[i + 1 :]:
            overlap = a[1] < b[2] and b[1] < a[2]
            related = a[3] == b[4] or b[3] == a[4]
            if overlap and not related:
                pairs.append((a[0], b[0]))
    return pairs

If that function returns a pair, any narrative that says "then" between those two tools is unsupported. Stop asking the model which file it edited first. The span tree already answered: both, or neither, with no order. Git is the same shape. A merge commit message is not a linear history until you pick a topology. Logs are the message. Spans are the graph.

The debug order that stays honest is therefore short, and it is the opposite of how terminals train you. Diff closed tool spans against the previous green run. Inspect concurrent pairs before you trust any "then." Only after those two steps read the model summary, and only as a hypothesis. The terminal shows words first. The span file is quieter. Quiet is the point. Words that cannot join a span were never observations.

Where a dump host actually helps

You still need an agent that emits the two files. A laptop loop mixes local clock noise with remote tool time, and it often skips the side file because the terminal looked sufficient.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option matter here only as a way to run the agent so spans.jsonl and tools.log land on one host. The checker does not depend on that host. If you already have traces, skip the product. If you do not, a free server is scaffolding for the dump, not a substitute for the join key, and a UI summary is not a span. Export NDJSON. Run the gate. Then read anything written in English.

Who should not use this gate

The loop will not tell you the model was wrong. Incomplete traces and wrong answers are different bugs. Mixing them produces a session that oscillates between adding more logs and changing the prompt, which feels like progress and measures nothing.

It will not work for agents that stream unstructured prose as their only tool record. If you cannot add tool_call_id to the child process, stop. The promotion test will fail every run. That result is correct. It is also useless as a daily gate, the way a smoke alarm that cannot be silenced is useless as a ship checklist.

It will not reconstruct payloads you redacted. Promote ids and status codes. Leave secrets in a filter that runs before the log file exists. Strip the id and you have manufactured leftovers on purpose. Teams that sample traces at ingest should not use the method either. Sampling drops one side of a join by design. A partial sample of an agent that made a dozen tool calls is not a shorter story. It is a different, unnamed agent.

The same warning applies to tracers that buffer events until process exit and then die with the child. A killed tool produces leftovers. That is a reliability bug in the tracer. It is not a model-quality signal, and treating it as one will send you back into the prompt.

Hang the checker on CI if you hang anything. The chat transcript will still be there after the gate fails. Waiting does not make it more true.

A Laptop Clock Cannot Order Remote Agent Spans

Sam Sun — Tue, 22 Sep 2026 15:51:30 +0000

A laptop clock cannot order spans that were recorded on a remote agent host. Sort those records by trace_id and span_id, not by wall time. Treat every timestamp as an annotation from a clock domain you have not yet proven equal.

The failure mode is a false order. A local log line prints a tool payload at 12:00:01.410Z. The remote span for the same call records start_time=12:00:03.002Z. A time-sorted view now shows the result before the request. The model is not traveling backward. Two oscillators are.

This split is ordinary once generation and tools leave the debugger process. Free model access on a free server is enough to create it. The collector sits beside the agent. The engineer tails stdout on a laptop. Both sides stamp events with whatever that operating system currently believes is UTC.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode matters here only because its free model access and free server option place the generation hop on a host the laptop does not control. The join procedure below does not depend on that product. It depends on identifiers surviving the hop.

Time is not the graph

A trace is an identifier graph. Time is a measurement taken by a clock. Mixing them produces three artifacts that get filed as model bugs: negative durations, overlapping spans that were marked exclusive, and child spans that appear to start before their parents.

Call the remote host clock C_r and the laptop clock C_l. For any shared event e that both sides observe, skew is C_r(e) - C_l(e). If that quantity is not stable across the length of one run, duration math that subtracts a laptop stamp from a remote stamp is undefined. Concurrent tool calls make the damage visible because their true order is the call graph, not the arrival time of log lines on a USB-C cable.

The reusable rule is narrow. Join first on W3C traceparent fields. Compute skew only from events that carry both a local receipt time and a remote span timestamp. Refuse to rank spans by C_l until the measured skew is smaller than the smallest span you actually care about. That last clause is the one dashboards skip.

W3C traceparent is a 55-character contract, not a log flavor. Version 00, a 32-hex trace, a 16-hex parent span, and a 2-hex flag field. If a proxy truncates the header, or mints a fresh span per hop and forgets to copy the trace, the laptop and the server still have clocks. They no longer have a join key. Repair the propagator before arguing about latency.

A minimal shared schema

The records below are constructed. They are not sampled from a customer account and they are not a benchmark. They exist so a test can fail when someone sorts by time and calls the result causality.

# join_clocks.py — proposed join, not a measured production trace
from __future__ import annotations

from dataclasses import dataclass
from typing import Optional
import json
import re

TRACEPARENT_RE = re.compile(
    r"^[\t ]*([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})[\t ]*$"
)

@dataclass(frozen=True)
class LocalEvent:
    event_id: str
    trace_id: str
    span_id: str
    name: str
    recv_unix_ms: int  # laptop clock, epoch ms
    payload: str

@dataclass(frozen=True)
class RemoteSpan:
    trace_id: str
    span_id: str
    parent_span_id: Optional[str]
    name: str
    start_unix_ms: int  # remote host clock
    end_unix_ms: int
    status: str

def parse_traceparent(value: str) -> tuple[str, str]:
    m = TRACEPARENT_RE.match(value)
    if not m:
        raise ValueError(f"unusable traceparent: {value!r}")
    version, trace_id, span_id, flags = m.groups()
    if version != "00":
        raise ValueError(f"unsupported traceparent version {version}")
    return trace_id, span_id

def join_by_span(events: list[LocalEvent], spans: list[RemoteSpan]) -> list[dict]:
    index = {(s.trace_id, s.span_id): s for s in spans}
    rows = []
    for ev in events:
        span = index.get((ev.trace_id, ev.span_id))
        rows.append({
            "event_id": ev.event_id,
            "name": ev.name,
            "trace_id": ev.trace_id,
            "span_id": ev.span_id,
            "has_span": span is not None,
            "local_recv_ms": ev.recv_unix_ms,
            "remote_start_ms": None if span is None else span.start_unix_ms,
            "remote_end_ms": None if span is None else span.end_unix_ms,
            "remote_status": None if span is None else span.status,
            "skew_recv_minus_start_ms": (
                None if span is None else ev.recv_unix_ms - span.start_unix_ms
            ),
        })
    return rows

A time sort on local_recv_ms and a time sort on remote_start_ms will disagree as soon as skew exceeds the gap between two tool calls. The join above does not care. It keys the map with the only values both sides must copy: the 32-hex trace and the 16-hex span. Missing keys stay missing. They do not get “nearby in time” substitutes. Nearby in time is how a write from trace B gets blamed on trace A.

Export the remote side as spans, not as a screenshot. A JSONL dump with one closed span per line is enough. The laptop side is noisier. Stdout, stderr, and the HTTP client each have a clock. Stamp recv_unix_ms at the moment the process observes the line, and copy traceparent off the line if the runtime echoed it. If the runtime did not echo it, the local event is not in the graph. It is a diary entry.

Measure skew before ranking

Skew is not a single integer. It is a distribution over shared events. The procedure uses events the laptop can observe only after the remote span exists: a tool result that echoes span_id. Receipt-minus-start is not duration. Duration lives inside one clock. Receipt-minus-start is a travel-plus-skew number, and it is useful only as a sample in that distribution.

def skew_samples(rows: list[dict]) -> list[int]:
    out = []
    for row in rows:
        if row["remote_start_ms"] is None:
            continue
        out.append(row["local_recv_ms"] - row["remote_start_ms"])
    return out

def skew_report(samples: list[int]) -> dict:
    if not samples:
        return {"n": 0, "usable_for_order": False, "reason": "no shared events"}
    samples = sorted(samples)
    median = samples[len(samples) // 2]
    spread = samples[-1] - samples[0]
    return {
        "n": len(samples),
        "min_ms": samples[0],
        "median_ms": median,
        "max_ms": samples[-1],
        "spread_ms": spread,
        # Proposed threshold: if spread exceeds the smallest tool gap you debug, stop.
        "usable_for_order": spread < 50,
    }

def order_inversions(events: list[LocalEvent], spans: list[RemoteSpan]) -> list[dict]:
    """Flag pairs whose local order and remote order disagree."""
    joined = {(e.trace_id, e.span_id): e for e in events}
    named = [s for s in spans if (s.trace_id, s.span_id) in joined]
    inversions = []
    for i, a in enumerate(named):
        for b in named[i + 1:]:
            local_a = joined[(a.trace_id, a.span_id)].recv_unix_ms
            local_b = joined[(b.trace_id, b.span_id)].recv_unix_ms
            remote_cmp = (a.start_unix_ms > b.start_unix_ms) - (
                a.start_unix_ms < b.start_unix_ms
            )
            local_cmp = (local_a > local_b) - (local_a < local_b)
            if remote_cmp != 0 and local_cmp != 0 and remote_cmp != local_cmp:
                inversions.append({
                    "a": a.span_id,
                    "b": b.span_id,
                    "remote_order": "a_before_b" if remote_cmp < 0 else "b_before_a",
                    "local_order": "a_before_b" if local_cmp < 0 else "b_before_a",
                })
    return inversions

Read usable_for_order as a gate, not as a quality score. A 40 ms spread can still invert two 15 ms tool calls. If the debug question is which file write happened first, the identifiers on those write spans are the evidence. The clocks are commentary. Commentary that disagrees with the graph is how a retry gets described as “the model wrote twice at once.”

Parent-child checks belong on identifiers too. If tool_a.parent_span_id is not the generation span, the graph is broken even when every timestamp looks sorted. A pretty waterfall is not a propagator test. The waterfall can be drawn from bad times. The parent pointer cannot.

A constructed run

Two tool spans, one parent generation span, and three local log lines. The numbers are illustrative. They are chosen so a time-sorted laptop log lies.

def constructed_run() -> None:
    trace = "a" * 32
    parent = "b" * 16
    tool_a = "c" * 16
    tool_b = "d" * 16

    spans = [
        RemoteSpan(trace, parent, None, "generation", 1_000_000, 1_000_800, "ok"),
        RemoteSpan(trace, tool_a, parent, "read_file", 1_000_120, 1_000_200, "ok"),
        RemoteSpan(trace, tool_b, parent, "write_file", 1_000_150, 1_000_260, "ok"),
    ]
    events = [
        LocalEvent("e1", trace, parent, "generation_end", 1_000_640, "done"),
        LocalEvent("e2", trace, tool_a, "read_file_result", 1_000_090, '{"ok":true}'),
        LocalEvent("e3", trace, tool_b, "write_file_result", 1_000_310, '{"ok":true}'),
    ]

    rows = join_by_span(events, spans)
    print(json.dumps({
        "join": rows,
        "skew": skew_report(skew_samples(rows)),
        "inversions": order_inversions(events, spans),
    }, indent=2))

e2 arrives on the laptop 30 ms before the remote read_file start. A time-sorted local log therefore claims the result preceded the call. The join still attaches e2 to tool_a. The inversion list reports that local order and remote order disagree for tool_a versus parent. That object is the bug report. It is a clock report, not a prompt report.

The same constructed run shows why “exclusive time” across hosts is not exclusive. Generation on C_r overlaps the laptop’s receipt of write_file_result on C_l. Subtracting those stamps fabricates an untraced remainder that never happened. Keep exclusive-time math inside C_r. If the remote collector does not emit closed spans, you do not have exclusive time. You have an unfinished sentence.

A practical debug loop follows from that. Pull one failing trace_id from the remote export. Parse every local line that carries the same trace_id. Join. Print inversions and the skew spread. If inversions exist, stop talking about the model. If the spread is larger than the tool gap under study, stop talking about order. Only a clean join with a small spread is allowed to support a duration claim, and even then the duration must be computed from one side’s stamps.

# proposed local commands — not an executed session log
python join_clocks.py > join.json
python - <<'PY'
import json
d = json.load(open("join.json"))
print("inversions", len(d["inversions"]))
print("skew", d["skew"])
PY

A small regression test

The test is a proposal. It encodes the invariant: join stability under a constant skew, and a hard fail when ranking by the laptop clock would silently reorder work.

# test_join_clocks.py — proposed, not executed in this article
from join_clocks import (
    LocalEvent, RemoteSpan, join_by_span, order_inversions,
    skew_report, skew_samples,
)

def test_join_survives_constant_skew():
    trace, span = "1" * 32, "2" * 16
    spans = [RemoteSpan(trace, span, None, "tool", 5000, 5100, "ok")]
    events = [LocalEvent("x", trace, span, "tool_result", 5480, "ok")]
    rows = join_by_span(events, spans)
    assert rows[0]["has_span"] is True
    assert rows[0]["skew_recv_minus_start_ms"] == 480

def test_time_sort_is_not_the_gate():
    trace = "1" * 32
    a, b = "a" * 16, "b" * 16
    spans = [
        RemoteSpan(trace, a, None, "tool_a", 100, 150, "ok"),
        RemoteSpan(trace, b, None, "tool_b", 120, 180, "ok"),
    ]
    events = [
        LocalEvent("e_b", trace, b, "b_result", 90, "ok"),  # laptop saw b first
        LocalEvent("e_a", trace, a, "a_result", 200, "ok"),
    ]
    inv = order_inversions(events, spans)
    assert inv, "clock disagreement must surface as inversion, not as a silent reorder"
    report = skew_report(skew_samples(join_by_span(events, spans)))
    assert report["usable_for_order"] is False or inv

Run it with python -m pytest test_join_clocks.py -q after both files are on PYTHONPATH. The assertion is the point. A green test that sorts by recv_unix_ms and happens to match remote order on one lucky run is not coverage. Luck is how clock bugs survive until concurrent tools appear on a Tuesday.

What the join is allowed to answer

When a debug question is about identity — which tool, which file, which retry — answer from trace_id, span_id, and parent_span_id. When the question is about cost of time — queue wait versus generation versus tool — answer from a single clock domain. If the only timestamps on hand mix C_r and C_l, convert nothing. Export the remote spans as-is and ignore local wall time for ordering.

Question	Evidence that can answer it	Evidence that cannot
Did this tool run in this trace?	matching `trace_id` + `span_id`	log order on the laptop
Did write happen after read?	parent/child or explicit link, same clock	mixed-host timestamps
How long did generation take?	`end - start` on the remote span	local print times
Why is duration negative?	clock domain mix, not a model defect	prompt text

Negative duration is a clock symptom. Overlapping exclusive spans across hosts are the same symptom. Neither is a reason to change temperature, top-p, or the system prompt. Those knobs do not synchronize NTP.

Retry logic is the other place mixed clocks impersonate a model. A second tool span is a new span_id. It is not an in-place update of the first. If the laptop logs collapse both results onto one line because “the name was the same,” the join will look like a duplicate payload on one span. Keep the call id. Names are for humans. Identifiers are for order.

Limitations

The procedure assumes both sides copy traceparent without truncation. Some proxies keep trace_id and mint a new span_id per hop. That remains joinable, but the laptop event must record the span it actually received, not the span a comment in the SDK promised. Version bytes other than 00 should fail closed. A parser that strips unknown versions will invent joins.

It also assumes a shared event exists. A generation span with no echoed id in stdout cannot be skew-measured. You can still store the span. You cannot claim the local log is late or early. Silence is not latency.

NTP steps during a run invalidate the spread test. Monotonic clocks do not survive a host boundary. If a free server recycles the process between runs, do not reuse skew from the previous process. Measure per run. Then throw the number away. A saved skew from yesterday is a third clock, and it will lie with confidence.

Sampling is out of scope and still fatal. A parent span that survives when the child error span was dropped will look healthy on both clocks. The join cannot reconstruct a span that was never exported. Completeness is a collector problem. This script only reports holes as has_span: false.

This is the wrong tool when the agent and the debugger already share one process and one clock. It is also the wrong tool for token accounting, prompt quality, or filesystem diffs. Those questions need other traces. The join will not tell you whether a model is correct. It will tell you whether your ordering evidence is even in the same universe. Do not use the laptop clock to gate CI. A flaky order inversion will page the model, not the time daemon.

If that remote host is already a MonkeyCode free server in your loop, export one failing trace and run the join before editing the prompt. The identifiers are the check. The dashboard sort order is not.

An Agent Claim Is Open Until a Child Span Closes It

Sam Sun — Mon, 21 Sep 2026 14:08:42 +0000

An assistant transcript is a press release. A child span is a receipt. If the model says it ran the test suite and the trace has no child span for that tool under the generation that made the claim, the run is not done. The claim is still open, the same way a merge request stays open when CI never started.

Teams keep scoring agent runs from the last paragraph of the chat. That paragraph is generated text. It can describe a successful pytest invocation, a patched module, and a green exit code while the tool layer never executed. Walk the span tree instead of grading the prose, and the mismatch becomes mechanical.

Treat the generation span as a function call. Tool spans are the callees. A named tool inside that generation is an unresolved symbol until a child with a matching parent_id and tool name exists, reaches a terminal status, and, when the protocol provides it, carries the same tool_call_id. This is not line coverage of your application; it is claim coverage of the agent’s own statements.

The rest of this piece is a local checker and a fixture. The checker reads JSON, builds a parent map, extracts claims from generation spans, and prints every claim that never closed. It does not need a vendor UI, and it does not need you to trust the model’s summary.

A usable span is boring on purpose. Give it an id, an optional parent_id, a name, start and end timestamps in milliseconds, a status, and a small attribute bag. Put the assistant’s claimed tools on the generation span as structured objects, not as a sentence to be regexed later. If your exporter already follows OpenTelemetry-style GenAI fields, map those names in; the invariant does not care which SDK wrote the file.

#!/usr/bin/env python3
"""claim_span_coverage.py

Close an agent claim only when a child tool span exists.

A claim is a (generation_span_id, tool_name, tool_call_id|None) tuple
taken from the generation span's `claimed_tools` attribute.
A child closes the claim when:
  - child.parent_id == generation_span_id
  - child.name == tool_name
  - child.status in {"ok", "error"}  # error still counts as observed
  - if tool_call_id is present on both, they match

This file is meant to be run locally against the fixture below.
It performs no network I/O.
"""
from __future__ import annotations

import json
import sys
from collections import defaultdict
from typing import Any

TERMINAL = {"ok", "error"}


def load_trace(path: str) -> dict[str, Any]:
    with open(path, encoding="utf-8") as handle:
        payload = json.load(handle)
    if not isinstance(payload.get("spans"), list):
        raise ValueError("trace must contain a spans array")
    return payload


def index_spans(spans: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
    by_id: dict[str, dict[str, Any]] = {}
    for span in spans:
        span_id = span.get("id")
        if not span_id or span_id in by_id:
            raise ValueError(f"missing or duplicate span id: {span_id!r}")
        by_id[span_id] = span
    return by_id


def children_of(spans: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
    tree: dict[str, list[dict[str, Any]]] = defaultdict(list)
    for span in spans:
        parent = span.get("parent_id")
        if parent:
            tree[parent].append(span)
    return tree


def claims_from(span: dict[str, Any]) -> list[dict[str, Any]]:
    raw = span.get("attributes", {}).get("claimed_tools", [])
    claims = []
    for item in raw:
        if isinstance(item, str):
            claims.append({"name": item, "tool_call_id": None})
        elif isinstance(item, dict) and item.get("name"):
            claims.append(
                {
                    "name": item["name"],
                    "tool_call_id": item.get("tool_call_id"),
                }
            )
    return claims


def coverage_report(trace: dict[str, Any]) -> dict[str, Any]:
    spans = trace["spans"]
    by_id = index_spans(spans)
    tree = children_of(spans)
    open_claims = []
    closed_claims = []
    orphans = []

    for span in spans:
        parent_id = span.get("parent_id")
        if parent_id and parent_id not in by_id:
            orphans.append(span["id"])

    for span in spans:
        for claim in claims_from(span):
            matched = None
            for child in tree.get(span["id"], []):
                if child.get("name") != claim["name"]:
                    continue
                if child.get("status") not in TERMINAL:
                    continue
                child_cid = child.get("attributes", {}).get("tool_call_id")
                if (
                    claim["tool_call_id"]
                    and child_cid
                    and claim["tool_call_id"] != child_cid
                ):
                    continue
                matched = child
                break
            record = {
                "generation_id": span["id"],
                "tool": claim["name"],
                "tool_call_id": claim["tool_call_id"],
            }
            if matched is None:
                open_claims.append(record)
            else:
                record["child_id"] = matched["id"]
                record["child_status"] = matched.get("status")
                closed_claims.append(record)

    return {
        "closed": closed_claims,
        "open": open_claims,
        "orphans": orphans,
        "ok": not open_claims and not orphans,
    }


def main() -> int:
    if len(sys.argv) != 2:
        print(
            "usage: python3 claim_span_coverage.py fixture.json",
            file=sys.stderr,
        )
        return 2
    report = coverage_report(load_trace(sys.argv[1]))
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if report["ok"] else 1


if __name__ == "__main__":
    raise SystemExit(main())

Save a fixture that contains one honest closed claim and one dangling generation. The dangling row is the interesting case: the model named run_tests, and nothing under that parent ever ran.

{
  "trace_id": "run-2026-09-21-claim-coverage",
  "spans": [
    {
      "id": "gen-1",
      "parent_id": null,
      "name": "generation",
      "start_ms": 0,
      "end_ms": 40,
      "status": "ok",
      "attributes": {
        "claimed_tools": [
          {"name": "read_file", "tool_call_id": "c1"}
        ]
      }
    },
    {
      "id": "tool-1",
      "parent_id": "gen-1",
      "name": "read_file",
      "start_ms": 12,
      "end_ms": 18,
      "status": "ok",
      "attributes": {"tool_call_id": "c1", "path": "src/app.py"}
    },
    {
      "id": "gen-2",
      "parent_id": null,
      "name": "generation",
      "start_ms": 41,
      "end_ms": 90,
      "status": "ok",
      "attributes": {
        "claimed_tools": [
          {"name": "run_tests", "tool_call_id": "c2"}
        ]
      }
    }
  ]
}

Wire the two files and run the checker as a command, not as a dashboard screenshot.

python3 claim_span_coverage.py fixture.json
echo "exit=$?"

Expected stdout on this fixture is a JSON object with read_file under closed, run_tests under open, an empty orphans list, and ok set to false. The process exit code is 1. That exit code is the gate: a CI step can refuse to label the agent run green when a claim never grew a child.

Error status still closes the claim. An error child means the tool was observed, not that the user task succeeded. If you need task success, inspect the tool payload in a later gate. Mixing the two checks is how teams treat a missing pytest span and a failing pytest span as the same incident. They are not.

Orphan spans are the other refusal. A tool row whose parent_id points at a span that is not in the file is a side effect that escaped the DAG. Subprocess helpers do this when the agent forks a script and the script logs to a different tracer. The work may have happened on disk. It did not happen inside this trace, so this trace cannot close the claim.

Clock order is a weak substitute for parent_id. Two spans that overlap in wall time might be concurrent callees, or they might be a generation talking about work that already finished under a previous parent. The checker ignores start_ms except as documentation. If you later add a causality assertion, require child.start_ms >= parent.start_ms and child.end_ms <= parent.end_ms only after you know the exporter uses inclusive envelopes. Many runtimes close the generation before the last tool callback is flushed. That pattern produces false open claims unless you buffer tool exits before you emit the parent end event.

The export wrapper is the other half of the workflow. Record the tool name and id on the generation span as soon as the model returns a structured tool-call object, start the child in that same context, and flush children before ending the parent. The following is a labeled sketch, not a drop-in SDK.

# Proposal: attach claims at dispatch time so the checker is not parsing prose.
# Adapt field names to your runtime. This block is unexecuted.

def on_model_tool_calls(generation, tool_calls):
    generation.attributes.setdefault("claimed_tools", [])
    for call in tool_calls:
        generation.attributes["claimed_tools"].append(
            {"name": call.name, "tool_call_id": call.id}
        )

def run_tool(generation, call, dispatch):
    child = generation.start_child(name=call.name)
    child.attributes["tool_call_id"] = call.id
    try:
        payload = dispatch(call)
        child.status = "ok"
        return payload
    except Exception:
        child.status = "error"
        raise
    finally:
        child.end()

A conservative parser over free text is a last resort. Prose such as “you could run pytest” will over-claim. Prefer the model API’s tool-call objects when they exist. Structured objects plus child spans are the smallest honest picture of an agent run.

Token counts, file diffs, and summary embeddings do not substitute for this gate. A cheap completion can emit “tests passed” in a handful of tokens. Two summaries can be close in embedding space while only one run actually executed run_tests. Presence of a terminal child span is a cheaper and stricter signal than either of those proxies.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The coverage file is host-agnostic. If you need a machine to wrap an agent, export this JSON, and iterate without first provisioning your own box, MonkeyCode is an open-source project that currently offers free model access and a free server option. Use that path only as a collector. Keep the checker and the fixture in your own repo so the invariant survives any host.

The approach has sharp edges. Sampling that drops tool spans will invent open claims. Truncated exports that omit children will do the same. Encrypted or redacted tool names will fail the name match even when the child exists. Multi-agent runs that do not share a trace_id will look like two empty forests. None of those failures mean the agent was idle. They mean the evidence was incomplete, and incomplete evidence cannot close a claim.

Do not use this gate as a proof of semantic correctness. A write_file child with status ok does not prove the bytes on disk are the bytes you wanted. Do not use it as an authorization log, a billing meter, or a safety interlock for anything that moves money, rotates credentials, or actuates hardware. Those domains need signed tool results and an allowlist, not a parent pointer.

Skip the whole pattern if your runtime cannot emit parent links at all. A flat log of tool names without ids is not a DAG, and forcing it into this checker will only print noise. Fix context propagation first. Then turn the missing-child case into a failing exit code.

The reusable debug loop is short. Export the trace. Run the checker. For every open claim, either emit the missing child, delete the claim from the generation, or fail the run. For every orphan, repair propagation before you debate the model. Keep the failing fixture next to the prompt that produced it. The next regression will be another missing child, not a new narrative in the chat window.

Unfingerprinted Generation Spans Cannot Be Diffed

Sam Sun — Sun, 20 Sep 2026 12:02:51 +0000

A trace diff is undefined until every generation span carries a fingerprint of the prompt, the tool schema, and the declared model identity. Output tests do not supply that. Neither does a green exit code.

When an agent run looks worse overnight, the usual reflex is to rewrite the system prompt. That reflex assumes the two runs were the same experiment. They often were not. Shared free-model endpoints omit model, rewrite routing, or inject a provider preamble you never hashed. The second run is a different instrument. Comparing it to the first is like subtracting voltages measured on two unlabeled probes.

The measurement problem is older than agents. A unit test that asserts on the final file will pass while the model retried a tool four times, dropped a call_id, or answered from a different checkpoint than yesterday. Process and product diverged. If you only keep the product, you cannot say which one moved.

A generation fingerprint is a small, stable tuple stored as span attributes. It is not a dashboard slogan. Four fields are enough to make two traces comparable, or to prove they are not.

prompt.hash is a truncated SHA-256 of canonical JSON: system text, user text, and the tool JSON schema. tool.schema_hash is the same hash over the tools array alone, so a prompt-only edit does not look like a tool-surface change. gen_ai.request.model is the declared identifier if the exporter emitted one. If it is missing, the span is incomparable by default. gen_ai.request.temperature belongs in the tuple because a silent 0.0 to 0.7 swap will scatter tool-call order without touching the prompt.

Hash canonicalization must be boring and strict. Sort keys. Reject NaN. Encode UTF-8. Do not pretty-print. Two semantically equal prompts that differ by a trailing newline are different experiments, and the hash should say so.

The following module is a template. Wire it to your own span exporter. It has not been executed against a public corpus in this article.

# fingerprint.py — template, not a vendor SDK
from __future__ import annotations

import hashlib
import json
from typing import Any

def canonical(obj: Any) -> bytes:
    return json.dumps(
        obj,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
        allow_nan=False,
    ).encode("utf-8")

def short_sha(obj: Any, n: int = 16) -> str:
    return hashlib.sha256(canonical(obj)).hexdigest()[:n]

def generation_fingerprint(
    *,
    system: str,
    user: str,
    tools: list[dict],
    model: str | None,
    temperature: float | None,
) -> dict[str, str]:
    prompt_payload = {"system": system, "user": user, "tools": tools}
    attrs = {
        "prompt.hash": short_sha(prompt_payload),
        "tool.schema_hash": short_sha(tools),
        "fingerprint.missing_model": "true" if not model else "false",
    }
    if model:
        attrs["gen_ai.request.model"] = model
    if temperature is not None:
        attrs["gen_ai.request.temperature"] = str(temperature)
    attrs["fingerprint.tuple"] = short_sha(attrs)
    return attrs

Attach those attributes on the generation span at start, not after the model returns. A fingerprint written on the way out can include the completion and will then change when the model is flaky. You want the request identity, not the answer identity. Completion hashes belong on a child event if you need them.

Free shared endpoints make the missing-model case common. The HTTP layer answers. The span does not name a checkpoint. Re-running the same prompt on another host will not repair that. Refuse comparability until the exporter records an identifier, or pin one yourself from configuration rather than from the response body. Reading the model name back from the completion is how silent swaps leak into “stable” traces.

# stamp_model_from_config.py — template
import json
import os
import sys

declared = os.environ.get("AGENT_MODEL_ID", "").strip()
if not declared:
    print("AGENT_MODEL_ID unset; refusing export", file=sys.stderr)
    sys.exit(2)

for line in sys.stdin:
    span = json.loads(line)
    if span.get("kind") == "generation" and not span.get("gen_ai.request.model"):
        span["gen_ai.request.model"] = declared
        span["fingerprint.missing_model"] = "false"
        span["fingerprint.model_source"] = "config"
    sys.stdout.write(json.dumps(span, ensure_ascii=False) + "\n")

export AGENT_MODEL_ID=from-config-not-from-response
python stamp_model_from_config.py < raw.ndjson > stamped.ndjson

fingerprint.model_source=config is an admission, not a boast. It records that the name did not come from the provider. Downstream diffs can still run. They must not pretend the checkpoint was observed.

Export two runs as JSON-lines, one span per line. The comparer below does not grade quality. It answers a narrower question: may these traces be diffed at all?

# compare_fingerprints.py — template
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = ("prompt.hash", "tool.schema_hash")

def load_generations(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        span = json.loads(line)
        if span.get("kind") != "generation":
            continue
        rows.append(span)
    return rows

def comparable(a: dict, b: dict) -> tuple[bool, str]:
    if a.get("fingerprint.missing_model") == "true" or b.get("fingerprint.missing_model") == "true":
        return False, "missing_model"
    keys = REQUIRED + ("gen_ai.request.model", "gen_ai.request.temperature")
    for key in keys:
        if a.get(key) != b.get(key):
            return False, f"mismatch:{key}"
    return True, "ok"

def main(path_a: str, path_b: str) -> int:
    gens_a = load_generations(Path(path_a))
    gens_b = load_generations(Path(path_b))
    if not gens_a or not gens_b:
        print("no generation spans", file=sys.stderr)
        return 3
    # Index alignment is a first pass. Prefer generation_id in production.
    n = min(len(gens_a), len(gens_b))
    incomparable = 0
    for i in range(n):
        ok, why = comparable(gens_a[i], gens_b[i])
        if not ok:
            incomparable += 1
            print(f"span[{i}] incomparable ({why})")
    if len(gens_a) != len(gens_b):
        print(f"generation count {len(gens_a)} != {len(gens_b)}")
        return 2
    if incomparable:
        return 2
    print("fingerprints align; a path diff is defined")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run it as a gate in front of any path diff you already keep.

python compare_fingerprints.py run-a.ndjson run-b.ndjson
echo $?   # 0 defined, 2 incomparable, 3 missing generations

Exit code 2 is not a product failure. It is a lab-setup failure. Do not open a prompt PR. Do not bump a temperature. Restore the fingerprint first.

The decision rule is small enough to keep in the comparer, not in a wiki. If fingerprint.missing_model is true on either side, the traces are incomparable. If prompt.hash differs, you changed the experiment; attribute the delta to the prompt, not to “the agent.” If only tool.schema_hash differs, the model saw a different action surface. If only gen_ai.request.model differs, you are looking at a routing change. If the tuple matches and the child tool graph does not, then—and only then—you have an agent-path delta worth debugging.

That last case is the only one in which a span-level path diff has meaning. Everything else is a contaminated A/B test. Teams that skip the gate will spend the week “fixing” a prompt that never moved.

If you need a second machine to collect the B run, a free server option is enough to host the exporter and the comparer. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Treat both as a re-run host for the fingerprint gate above, not as a substitute for the attributes. After the spans exist, the method does not depend on a particular product.

Limitations are sharp. A hash does not see a provider-side system preamble that never entered your canonical JSON. Two fingerprints can match while safety filters rewrite the completion. Temperature in the tuple does not capture nucleus sampling or frequency penalty if you never record them. Index alignment of generation spans is a weak join. Production code should join on a client-generated generation_id. The templates above are a specification for exit codes, not a benchmark.

Do not store raw prompts next to the hashes if those prompts contain secrets. The hash is for equality, not for recovery. Do not use the fingerprint as a billing key. Do not sample generation spans before hashing. A 1% sample will drop the only span whose model identity was missing and will fake comparability.

Skip this workflow if every run already pins a model digest, a signed prompt bundle, and a tool-schema version in CI. You already have a stronger fingerprint, and SHA-16 on the side is noise. If you have no traces yet, record closed generation spans first. Hashing empty files will only certify that two voids are equal. If the agent is a single non-tool completion graded by a human, a prompt hash plus an output rubric is enough and the comparer is overhead.

The reusable debug loop is then four beats, always in this order. Fingerprint. Refuse the diff when the tuple disagrees. Diff tool children only after a defined comparison. Only then touch the prompt. Reverse the order and you will keep shipping prompt edits that chase routing noise.

If you already export spans, add the four attributes before you add another chart.

Join Tool Exits to Spans Before Trusting the Summary

Sam Sun — Sat, 19 Sep 2026 10:08:21 +0000

A model’s closing paragraph is not a trace. The only agent run you can debug is the one where every tool process exit inner-joins to a closed span on a caller-issued invocation_id, and where span status is a function of that exit code rather than of later assistant text.

That rule sounds pedantic until a coding agent shells out twice, retries once, and then writes “ran the tests.” The summary is one sentence. The machine did three things. Two of them may have failed. If those failures never became spans, the debug loop starts from fiction.

Think of baggage tags. A passenger name on a boarding pass does not locate a suitcase. The tag number does. Tool names in a chat transcript are passenger names. invocation_id is the tag. Order of appearance in the log is not a join key. Concurrent tools scramble wall-clock order, and retries mint extra exits that never show up in the last message.

What the join actually asserts

Treat the run as two relations that must agree. Relation E is process exits: one row per OS-level tool invocation, with invocation_id, exit_code, ended_at, and a hash of stdout plus stderr. Relation S is closed spans: one row per span that claimed to wrap a tool, with the same invocation_id, a mapped span_status, and closed_at.

A run is join-complete when three predicates hold. Every row in E matches exactly one row in S. Every row in S matches exactly one row in E. For each matched pair, span_status equals the mapping of exit_code (zero to ok, anything else to error). Fail the run on leak, orphan, or mismatch. Do not “warn and continue.” A warning is how unmatched exits become folklore.

The mapping is deliberately crude. A test harness that prints FAILED and still exits 0 is a product bug in the tool, not a reason to let the model overwrite span status. If you need richer outcomes, put them on span events. Status stays a function of the process.

A proposed local harness

The checker below is a proposed local harness, not a production trace backend. It writes JSONL. It does not sample. Label it as unexecuted in your environment until you run it against a real agent wrapper.

# join_exits.py — proposed checker, file-backed JSONL
from __future__ import annotations

import hashlib, json, os, subprocess, time, uuid
from pathlib import Path

TRACE = Path(os.environ.get("TRACE_JSONL", "run.jsonl"))

def _write(row: dict) -> None:
    row["ts"] = time.time()
    with TRACE.open("a", encoding="utf-8") as f:
        f.write(json.dumps(row, sort_keys=True) + "\n")

def map_status(exit_code: int) -> str:
    return "ok" if exit_code == 0 else "error"

def run_tool(argv: list[str]) -> int:
    invocation_id = str(uuid.uuid4())
    env = os.environ.copy()
    env["INVOCATION_ID"] = invocation_id
    _write({"type": "span_open", "invocation_id": invocation_id, "argv": argv})
    proc = subprocess.run(argv, env=env, capture_output=True)
    payload = proc.stdout + b"\0" + proc.stderr
    digest = hashlib.sha256(payload).hexdigest()
    status = map_status(proc.returncode)
    _write({
        "type": "process_exit",
        "invocation_id": invocation_id,
        "exit_code": proc.returncode,
        "stdout_stderr_sha256": digest,
        "bytes": len(payload),
    })
    _write({
        "type": "span_close",
        "invocation_id": invocation_id,
        "span_status": status,
        "exit_code": proc.returncode,
    })
    return proc.returncode

def load_rows() -> list[dict]:
    if not TRACE.exists():
        return []
    return [json.loads(line) for line in TRACE.read_text().splitlines() if line.strip()]

def join_report(rows: list[dict]) -> dict:
    exits = {r["invocation_id"]: r for r in rows if r.get("type") == "process_exit"}
    closes = {r["invocation_id"]: r for r in rows if r.get("type") == "span_close"}
    leaks = sorted(set(exits) - set(closes))
    orphans = sorted(set(closes) - set(exits))
    mismatches = sorted(
        i for i in set(exits) & set(closes)
        if closes[i]["span_status"] != map_status(exits[i]["exit_code"])
    )
    return {
        "exit_count": len(exits),
        "close_count": len(closes),
        "leaks": leaks,
        "orphans": orphans,
        "status_mismatches": mismatches,
        "join_ok": not (leaks or orphans or mismatches),
    }

if __name__ == "__main__":
    import sys
    cmd = sys.argv[1:]
    if cmd[:1] == ["--report"]:
        report = join_report(load_rows())
        print(json.dumps(report, indent=2))
        raise SystemExit(0 if report["join_ok"] else 2)
    if not cmd:
        raise SystemExit("usage: join_exits.py <tool> [args...] | --report")
    raise SystemExit(run_tool(cmd))

Wrap the agent’s tool runner so it cannot exec except through run_tool. A one-line shell stand-in is enough to prove the join on a laptop:

export TRACE_JSONL="$PWD/run.jsonl"
: > "$TRACE_JSONL"
python join_exits.py python -c "print('ok')"
python join_exits.py python -c "raise SystemExit(1)"
python join_exits.py --report

The second command must produce span_status of error. If your agent later prints “all good,” the report still fails. That is the point. The summary is not a source row in either relation.

How summaries lie without leaking bytes

Three failure modes show up constantly, and none of them require a sophisticated model. The agent retries a formatter, the first process exits 1, the second exits 0, and only the second is wrapped in a span. Relation E has two rows. Relation S has one. The leak is the first exit. The summary mentions a single format step because language models compress.

The inverse also happens. A span is opened for pytest, the process is never started, and the span is closed ok because the orchestrator timed out of the thought loop and needed a status. That is an orphan. It is worse than a leak. A leak hides work. An orphan invents work.

Status mismatch is quieter. The process exits 2. A post-processor, or the model itself, writes span_status: ok because tests were “mostly fine.” The inner join on invocation_id succeeds. The run still has to fail. Id equality is not semantic equality.

If you want a regression hook, do not grep the assistant message. Assert the report:

python join_exits.py --report
test "$(python -c 'import json,sys; print(json.load(sys.stdin)["join_ok"])' < <(python join_exits.py --report))" = True

Better: read the JSON. Gate CI on join_ok, exit_count, and a ceiling on error closes. A bound on tool exits is an observability SLO. It is not a token budget. Token counters live in another pipeline and do not prove that a process was traced.

Where a free model host fits, and where it does not

The join does not need a particular vendor. It needs a wrapper you control. When the loop also needs a model to propose the next tool, standing up paid inference just to debug tracing is a poor trade. MonkeyCode is an open-source project that currently offers free model access and a free server option for that kind of harness work. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep secrets off that path. The checker hashes stdout and stderr; it does not redact them at the source. If a tool prints a token or a customer fixture, the JSONL becomes a credential store. Run the free server option only on synthetic repos and fixtures you would paste into a ticket. The join logic itself should stay local, next to the working tree.

Strip the product out and the method is unchanged. You still emit span_open, process_exit, and span_close. You still fail on leaks. That is the usefulness test.

Limitations, and who should skip this

This join assumes tools are processes. In-process function calls that never exec will not appear in relation E unless you fabricate an exit row. Fabricated exits are how orphans are born. If your agent’s tools are Python callables, instrument the callable, not a fake subprocess.

It also assumes one exit per invocation_id. Pipelines that reuse an id across retries will collapse two exits into one dict key in the sample checker. Retries need new ids. Correlate them with a parent_invocation_id field if you must, but do not recycle the primary key.

Do not use this as a substitute for OpenTelemetry if you already have a collector, span processors, and identity at the RPC layer. Dual-writing JSONL and OTLP without a single id space creates a second forest. Do not use it on jobs whose stdout is legally sensitive. Hashing does not erase the bytes you wrote before the hash.

Clock skew is out of scope. The join key is the id, not ended_at ≈ closed_at. If you add a time window as a hint, keep it diagnostic. Never let a 50ms skew decide that an exit “probably” belongs to a span with a different id.

The last assistant message can still be useful after the join passes. Read it as commentary. Commentary is not a row. If the join fails, do not debate the commentary. Fix the wrapper until leaks, orphans, and status_mismatches are empty lists, then read the prose.

If your traces are already JSONL, run join_exits.py --report on the last failed job before you add another log line. Extra logs without a join key are more passenger names. They will not find the bag.

Wall-Clock Minus Exclusive Spans Is the Untraced Remainder

Sam Sun — Fri, 18 Sep 2026 08:35:22 +0000

An agent run that lasts eight seconds is not an eight-second tool. The number that should gate a retry, a budget, or a claim that the loop did engineering work is the untraced remainder: wall-clock duration minus the union of closed spans, after children have been subtracted from parents. Everything else is a mood.

Most debug loops still print elapsed time and move on. That figure mixes model wait, overlapping tools, process startup, and gaps where the tracer was not listening. Treat it as cost and you will “optimize” a parallel read that was already cheap, or miss a two-second hole where the model sat idle with no span at all.

Inclusive duration is a nested Russian doll. A parent span that wraps a plan step and three tools will report the whole doll. Sum those parents with their children and you double-count the inner work. Parallel tools make it worse: two 400 ms reads that overlap are not 800 ms of exclusive machine time. They are closer to 400 ms of wall and 800 ms of billed tool occupancy, and those are different questions.

A restaurant bill is a fair analogy. Summing dish prices ignores tax, shared plates, and the twenty minutes you waited for a table. Wall-clock is the evening. Inclusive span sum is every dish counted with the table time glued on. Exclusive span math is the only way to see what each course actually occupied.

The contract is small. Every span must have a start, an end, a parent id or a root, and a kind that is either model or tool. If end_ms is missing, the run is not closed. If two siblings overlap, their inclusive times must not be added. If the union of all intervals is much smaller than the process lifetime, the remainder is untraced work, not a rounding error.

The artifact below is a single-file judge. It is a proposal you can run locally; it is not a production tracer. Feed it a JSON run, and it prints exclusive durations, overlap pairs, and the untraced remainder as a fraction of wall-clock.

#!/usr/bin/env python3
"""exclusive_span.py — attribute agent time after exclusive span math."""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class Span:
    span_id: str
    parent_id: str | None
    kind: str
    name: str
    start_ms: int
    end_ms: int

    @property
    def inclusive_ms(self) -> int:
        return self.end_ms - self.start_ms


def load_run(path: str) -> tuple[int, int, list[Span]]:
    with open(path, encoding="utf-8") as fh:
        raw = json.load(fh)
    spans = []
    for row in raw["spans"]:
        if row.get("end_ms") is None:
            raise ValueError(f"unclosed span {row.get('span_id')}")
        if row["end_ms"] < row["start_ms"]:
            raise ValueError(f"inverted span {row['span_id']}")
        spans.append(
            Span(
                span_id=row["span_id"],
                parent_id=row.get("parent_id"),
                kind=row["kind"],
                name=row["name"],
                start_ms=int(row["start_ms"]),
                end_ms=int(row["end_ms"]),
            )
        )
    return int(raw["started_at_ms"]), int(raw["ended_at_ms"]), spans


def children_of(spans: list[Span], parent_id: str) -> list[Span]:
    return [s for s in spans if s.parent_id == parent_id]


def exclusive_ms(span: Span, spans: list[Span]) -> int:
    kids = children_of(spans, span.span_id)
    covered = union_ms((k.start_ms, k.end_ms) for k in kids)
    clipped = 0
    for a, b in covered:
        lo = max(a, span.start_ms)
        hi = min(b, span.end_ms)
        if hi > lo:
            clipped += hi - lo
    return span.inclusive_ms - clipped


def union_ms(intervals: Iterable[tuple[int, int]]) -> list[tuple[int, int]]:
    ordered = sorted(intervals)
    if not ordered:
        return []
    out = [ordered[0]]
    for start, end in ordered[1:]:
        last_s, last_e = out[-1]
        if start <= last_e:
            out[-1] = (last_s, max(last_e, end))
        else:
            out.append((start, end))
    return out


def overlap_pairs(spans: list[Span]) -> list[tuple[str, str, int]]:
    pairs = []
    for i, a in enumerate(spans):
        for b in spans[i + 1 :]:
            if a.parent_id != b.parent_id:
                continue
            lo = max(a.start_ms, b.start_ms)
            hi = min(a.end_ms, b.end_ms)
            if hi > lo:
                pairs.append((a.span_id, b.span_id, hi - lo))
    return pairs


def remainder_ms(start: int, end: int, spans: list[Span]) -> int:
    covered = union_ms((s.start_ms, s.end_ms) for s in spans)
    covered_total = sum(b - a for a, b in covered)
    return max(0, (end - start) - covered_total)


def report(path: str) -> None:
    start, end, spans = load_run(path)
    wall = end - start
    rem = remainder_ms(start, end, spans)
    print(f"wall_ms={wall}")
    print(f"untraced_ms={rem} fraction={rem / wall if wall else 0:.3f}")
    for span in spans:
        print(
            f"{span.span_id} kind={span.kind} name={span.name} "
            f"inclusive={span.inclusive_ms} exclusive={exclusive_ms(span, spans)}"
        )
    for a, b, ms in overlap_pairs(spans):
        print(f"overlap {a} {b} ms={ms}")
    model_ex = sum(exclusive_ms(s, spans) for s in spans if s.kind == "model")
    tool_ex = sum(exclusive_ms(s, spans) for s in spans if s.kind == "tool")
    print(f"exclusive_model_ms={model_ex} exclusive_tool_ms={tool_ex}")
    if rem / wall > 0.25 if wall else False:
        print("FAIL: untraced remainder exceeds 25% of wall-clock")
        sys.exit(2)
    if tool_ex == 0 and wall > 0:
        print("FAIL: no exclusive tool time; this run is a chat, not a tool loop")
        sys.exit(3)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.stderr.write("usage: exclusive_span.py run.json\n")
        sys.exit(1)
    report(sys.argv[1])

A fixture makes the failure mode obvious. The process lives 8400 ms. Tools overlap. A 1800 ms gap has no span. Inclusive sums look busy. Exclusive math does not.

{
  "run_id": "demo-8400",
  "started_at_ms": 0,
  "ended_at_ms": 8400,
  "spans": [
    {"span_id": "m1", "parent_id": null, "kind": "model", "name": "plan",
     "start_ms": 50, "end_ms": 2100},
    {"span_id": "t1", "parent_id": "m1", "kind": "tool", "name": "read_a",
     "start_ms": 2200, "end_ms": 2650},
    {"span_id": "t2", "parent_id": "m1", "kind": "tool", "name": "read_b",
     "start_ms": 2300, "end_ms": 2700},
    {"span_id": "m2", "parent_id": null, "kind": "model", "name": "revise",
     "start_ms": 4500, "end_ms": 6200},
    {"span_id": "t3", "parent_id": "m2", "kind": "tool", "name": "patch",
     "start_ms": 6300, "end_ms": 6550}
  ]
}

Save the judge as exclusive_span.py and the fixture as run.json, then run it as a gate, not as a dashboard widget.

python3 exclusive_span.py run.json; echo exit:$?

On this fixture the wall is 8400 ms. The union of spans leaves a hole from 2700 to 4500, plus short edges at start and finish. Sibling tools t1 and t2 overlap by 350 ms, so adding their inclusive times invents 350 ms of serial work that never happened. Exclusive model time and exclusive tool time become the two columns you can actually budget. The script exits 2 because the untraced remainder is larger than a quarter of the run. That is the point.

A second command turns the same file into a regression check. Keep a known-good remainder ceiling next to the fixture. If a prompt change stretches the hole, the gate fails before anyone calls the agent “faster” because wall-clock happened to drop on a warm cache.

python3 - <<'PY'
import json, pathlib, subprocess, sys
raw = json.loads(pathlib.Path("run.json").read_text())
assert raw["ended_at_ms"] - raw["started_at_ms"] == 8400
proc = subprocess.run([sys.executable, "exclusive_span.py", "run.json"])
print("gate_status", proc.returncode)
PY

Use wall-clock when you care about user-visible wait. Use inclusive parent duration when you are debugging a single subtree and you want the doll, not the parts. Use exclusive duration when you allocate cost to a kind. Use the untraced remainder when you decide whether the tracer is complete enough to argue from. Mixing those four numbers is how teams convince themselves a chatty model is a fast toolchain.

The 25% remainder threshold in the script is a local policy, not a law. Tighten it on short, fully instrumented loops. Loosen it if your runtime injects GC pauses you do not span on purpose. Do not copy the constant into a vendor SLA. The useful part is the formula, not the cutoff.

Clock skew will lie. If tools run on another host and stamps are not on the same clock, overlap pairs become fiction and exclusive math subtracts the wrong children. Missing end_ms is worse: an open span makes inclusive duration undefined, so the script refuses the run rather than inventing a close. Async tools that finish after the parent has already closed will look like orphans; exclusive parent time will be too large and the remainder too small. A forest that is actually a DAG, with two parents claiming one child, is outside this model. Fix the shape first.

This debug loop is the wrong tool if you only have stdout lines, token counts, or a UI timer. It is also the wrong tool if your agent is a single model call with no tools; the second failure path will keep firing, and it should. Teams that need distributed tracing with baggage, sampling, and tail-based retention want an OpenTelemetry pipeline, not a hundred-line judge. The script exists to make a local run honest before you scale the lie.

Current public debate likes to separate “vibe” from engineering by tone. Duration attribution is blunter. A run whose exclusive tool time is zero did not touch the working tree in any spanned way. A run whose remainder is half the wall-clock did work you cannot replay. Neither case is fixed by a greener unit test that never saw the hole.

If you already emit closed spans from your own harness, the judge is enough. If you need a place to exercise the agent while you iterate on that harness, MonkeyCode’s free model access and free server option can host the loop so the remainder math stays local. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Ship the gate next to the fixture. When exclusive tool time rises and the untraced fraction falls, you have a trace you can budget. Until then, wall-clock is only how long you waited to be unsure.

Map Every Working-Tree Delta to a Closed Tool Span

Sam Sun — Thu, 17 Sep 2026 06:47:10 +0000

An agent run is finished when the working tree and the trace agree, not when the model emits a final sentence. git status --porcelain is the ledger. Closed tool spans are the journal entries. If a path changed and no span owns it, the run failed. If a span claims a write and the bytes are missing, the run failed. The summary paragraph is not evidence.

This gap is getting easier to miss. Public debate this month keeps returning to whether generated code is already “better than most developers,” and whether a green test suite is enough to call the session engineering. Those arguments skip a more primitive check. Tests observe outputs someone chose to assert. They do not observe every file the process touched. An agent can skip a tool, narrate the skip as success, and still pass a unit file that never opened the mutated path.

Think of a warehouse cycle count. The packing list the intern typed is the assistant message. The scanner at the dock is the trace. The pallets on the floor are git status. Nobody ships because the intern wrote “all packed.” You ship when every pallet id is on a scan record and every scan record points at a pallet that still exists. Agent tooling that treats the chat transcript as the dock scan will leak mutations into the next human commit.

The failure has two directions, and they are not symmetric. A ghost write is a dirty path with no closed span: a formatter, a shell redirect, a compiler, or a helper script that ran outside the traced adapter. A phantom write is a closed span whose after_hash is not on disk: the tool returned ok, the model thanked it, and a later step reverted the file, or the tool wrote a temp path and reported a source path. Either break should fail the run. A third, quieter break is a clean tree after a task that required a mutation. No-op success is still a false success if the prompt demanded an edit.

A reusable control is a bijection between path-level deltas and closed write spans. The harness below is a worked example, not a production trace backend. It records JSONL, hashes file bytes with SHA-256, and compares that set to git status --porcelain. It does not replace OpenTelemetry. It does claim that a run without this map is an anecdote.

Label the schema as an example. Each tool attempt appends one record. status must become closed before reconcile. op is write, delete, or read. Reads do not have to match porcelain. Writes and deletes do. path is repo-relative POSIX. before_hash and after_hash are hex or null.

# span_schema.py — example record shape, not a shipped SDK
import hashlib, json, time, uuid
from pathlib import Path

def sha256_file(path: Path):
    if not path.is_file():
        return None
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def open_span(trace_path, op, relpath, before_hash):
    rec = {
        "span_id": str(uuid.uuid4()),
        "ts_open": time.time_ns(),
        "op": op,
        "path": relpath,
        "before_hash": before_hash,
        "after_hash": None,
        "status": "open",
        "error": None,
    }
    with open(trace_path, "a", encoding="utf-8") as f:
        f.write(json.dumps(rec) + "\n")
    return rec["span_id"]

def close_span(trace_path, span_id, after_hash, error=None):
    rows = []
    with open(trace_path, encoding="utf-8") as f:
        for line in f:
            rec = json.loads(line)
            if rec["span_id"] == span_id:
                rec["ts_close"] = time.time_ns()
                rec["after_hash"] = after_hash
                rec["status"] = "error" if error else "closed"
                rec["error"] = error
            rows.append(rec)
    with open(trace_path, "w", encoding="utf-8") as f:
        for rec in rows:
            f.write(json.dumps(rec) + "\n")

Wrap every mutating tool. Do not wrap the model. The model can say anything. The adapter is the only process allowed to touch the tree. If the agent is given a raw shell, the shell itself must emit spans for every path it intends to change, or the later git diff will fail the run on purpose. That is the point. Untraced sed -i is not a clever escape. It is an open drawer.

# traced_write.py — example adapter
from pathlib import Path
from span_schema import sha256_file, open_span, close_span

def traced_write(repo: Path, trace_path: str, relpath: str, data: bytes):
    dest = (repo / relpath).resolve()
    if not str(dest).startswith(str(repo.resolve())):
        raise ValueError("path escapes repo")
    dest.parent.mkdir(parents=True, exist_ok=True)
    before = sha256_file(dest)
    span_id = open_span(trace_path, "write", relpath, before)
    try:
        dest.write_bytes(data)
        close_span(trace_path, span_id, sha256_file(dest))
    except Exception as exc:
        close_span(trace_path, span_id, sha256_file(dest), error=str(exc))
        raise

Collect the ledger from git, not from in-memory guesses. Porcelain is stable enough for a gate. Untracked files are mutations. Ignored files are a policy choice: if the agent is allowed to write node_modules, the bijection is already blind. Keep generated noise out of the repo, or accept that the map cannot see it.

# git_delta.py — example ledger
import subprocess
from pathlib import Path

def porcelain_paths(repo: Path) -> dict:
    out = subprocess.check_output(
        ["git", "status", "--porcelain", "-uall"],
        cwd=repo, text=True,
    )
    deltas = {}
    for line in out.splitlines():
        if not line:
            continue
        code, rest = line[:2], line[3:]
        if " -> " in rest:
            rest = rest.split(" -> ", 1)[1]
        deltas[rest.strip()] = code.strip()
    return deltas

Reconcile is a set comparison plus a hash check. Open spans fail first. Then every write or delete span must match a porcelain path. Then every porcelain path must match a span. Then after_hash must equal the bytes on disk for writes, and the path must be absent for deletes. Print the leftover sets. Do not summarize them in prose and call the run “mostly fine.”

# reconcile.py — example gate
import json, sys
from pathlib import Path
from span_schema import sha256_file
from git_delta import porcelain_paths

WRITE_OPS = {"write", "delete"}

def load_spans(trace_path):
    with open(trace_path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]

def reconcile(repo: Path, trace_path: str, require_mutation: bool) -> int:
    spans = load_spans(trace_path)
    open_ids = [s["span_id"] for s in spans if s.get("status") == "open"]
    if open_ids:
        print("FAIL open spans:", open_ids)
        return 2

    mutating = [
        s for s in spans
        if s.get("op") in WRITE_OPS and s.get("status") == "closed"
    ]
    disk = porcelain_paths(repo)
    span_paths = {s["path"] for s in mutating}
    ghost = sorted(set(disk) - span_paths)
    phantom = sorted(span_paths - set(disk))

    hash_miss = []
    for s in mutating:
        p = repo / s["path"]
        if s["op"] == "write" and sha256_file(p) != s.get("after_hash"):
            hash_miss.append(s["path"])
        elif s["op"] == "delete" and p.exists():
            hash_miss.append(s["path"])

    if require_mutation and not disk and not mutating:
        print("FAIL no-op tree on a mutating task")
        return 3
    if ghost or phantom or hash_miss:
        print("FAIL ghost:", ghost)
        print("FAIL phantom:", phantom)
        print("FAIL hash:", hash_miss)
        return 1
    print("PASS span-tree bijection")
    return 0

if __name__ == "__main__":
    repo = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    trace = sys.argv[2] if len(sys.argv) > 2 else "run.jsonl"
    require = "--require-mutation" in sys.argv
    sys.exit(reconcile(repo, trace, require))

The CI step should stay boring. That is the desired property. After the agent process exits, the job fails on any leftover path. Do not scan the final assistant message for “done” or “fixed.” Those strings are not a working tree.

python reconcile.py . run.jsonl --require-mutation

The following cases are unexecuted examples, not measured production results. They pin the three failure classes the gate is meant to catch.

# test_reconcile_cases.py — unexecuted examples
def test_ghost_write_fails(tmp_repo):
    # an untraced formatter touches extra.py; porcelain has it, spans do not
    assert reconcile(tmp_repo, "run.jsonl", False) == 1

def test_phantom_write_fails(tmp_repo):
    # a closed span claims src/app.py; git is clean
    assert reconcile(tmp_repo, "run.jsonl", False) == 1

def test_noop_on_required_mutation_fails(tmp_repo):
    assert reconcile(tmp_repo, "run.jsonl", True) == 3

Wire the agent so it can only call traced_write, a traced delete, and a traced shell that predeclares output paths. Isolation matters more than the hash function. A laptop already mid-refactor will fail the gate for human reasons: your own dirty files become ghosts. Run the agent in a clean clone. Keep the developer tree out of the ledger.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is a practical place to run that clone-plus-harness loop when the point is to keep the agent's tree off the developer's working copy. The scripts do not depend on it. They run against any model endpoint and any VM. The free options only remove the excuse that isolation needs a dedicated budget before the gate exists.

Limitations are sharp. The harness does not see mutations that never hit git, including ignored paths, Docker volume writes, and database rows. Line-ending conversion and formatters that rerun after the last span will produce hash misses even when the semantic edit is correct; trace the formatter or disable it during the run. Concurrent agents on one working tree break the bijection because porcelain cannot say which span owned a path. Case-insensitive filesystems alias paths the JSONL treats as distinct. Binary assets hash correctly but make traces noisy; keep them out of agent workspaces or accept large JSONL. This control also does not prove the change was right. It proves the change was accounted for. A well-traced, well-hashed, wrong patch still merges if tests are weak.

Do not use this approach for exploratory sessions where the human is supposed to edit the same tree. The gate will fight the human. Do not use it as a secrecy boundary. Hashes of file bytes are not redaction; secrets written into a traced file are still secrets in whatever store keeps the JSONL. Do not use it to rank models. A cheaper model that emits fewer ghost writes is not “better at coding.” It is better at staying inside the adapter. Those are different claims, and mixing them is how generated output gets called engineering.

The durable rule is small. Close every mutating span. Reconcile against porcelain. Fail on leftovers in either set. Until that map is closed, the agent did not finish, no matter how confident the last sentence reads.