DEV Community: Riley Wang

Tool Previews Lie. Store stdout_bytes and a Truncation Flag.

Riley Wang — Wed, 23 Sep 2026 16:29:07 +0000

A coding agent closed the flaky test ticket.
The JSONL span showed pytest and a short preview.
The next human run failed in the same module.

The wrapper had capped stdout at 8192 bytes.
The assertion text started after that cap.
The model never read the failing lines.

This is not a prompt problem first.
It is missing I/O metadata on the tool span.

The failure mode

Most agent traces store a short text preview.
They also store a loose success flag.
That pair cannot describe a clipped pipe.

Three wrappers hide the real tool result:

Head-cap keeps only the first N bytes.
Tail-cap keeps only the last N bytes.
Pipe-stall drops the stream and stores empty text.

The model then plans from a partial document.
Later debug sessions trust that partial document.
Teams retune temperature while the bytes were absent.

What typical spans already contain

A typical tool span already has a name.
It often includes arguments and a duration.
Some collectors also keep an exit code.

Those fields still miss clipped pipes.
Exit code 0 can sit beside truncated stdout.
Exit code 1 can sit beside truncated stderr.
Both streams can be empty after a full pipe.

Fields to add on every tool span

Record these keys on each tool span:

stdout_bytes: length before any wrapper cap
stderr_bytes: length before any wrapper cap
stdout_cap: wrapper limit in bytes, or null
stderr_cap: wrapper limit in bytes, or null
stdout_truncated: boolean after the cap check
stderr_truncated: boolean after the cap check
stdout_sha256: full-stream hash when unclipped
stderr_sha256: full-stream hash when unclipped
stdout_head_sha256 / stdout_tail_sha256: clipped case
encoding: utf-8 or binary

Keep the stored preview short on purpose.
Store hashes for bytes you refused to keep.
Never treat the preview as the full stream.

Proposed JSONL schema

The next record is a proposed schema only.
It is not sampled from a production fleet.
Use it as a contract for local collectors.

{
  "span_type": "tool",
  "tool_name": "pytest",
  "cwd": "/work/repo",
  "preview_stdout": "===== test session starts =====\n",
  "preview_stderr": "",
  "stdout_bytes": 24118,
  "stderr_bytes": 902,
  "stdout_cap": 8192,
  "stderr_cap": 8192,
  "stdout_truncated": true,
  "stderr_truncated": false,
  "stdout_sha256": null,
  "stderr_sha256": "b3a1...",
  "stdout_head_sha256": "9c21...",
  "stdout_tail_sha256": "44de...",
  "encoding": "utf-8",
  "ok_preview": true
}

ok_preview is what the agent saw.
It is not a claim about the full process.
Lint that distinction before you debug policy.

A local subprocess wrapper

The wrapper below is a labeled local example.
Run it against one command on your machine.
It is not a production sandbox or jail.

#!/usr/bin/env python3
"""Record truncation metadata around one tool command."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

CAP = 8192  # example cap; set from your collector
PREVIEW = 512


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def clip_meta(data: bytes, cap: int) -> dict:
    truncated = len(data) > cap
    kept = data[:cap] if truncated else data
    meta = {
        "bytes": len(data),
        "cap": cap,
        "truncated": truncated,
        "sha256": None if truncated else sha256_bytes(data),
        "head_sha256": sha256_bytes(data[: min(len(data), 256)]),
        "tail_sha256": sha256_bytes(data[-min(len(data), 256) :]) if data else None,
        "preview": kept[:PREVIEW].decode("utf-8", "replace"),
        "encoding": "utf-8",
    }
    return meta


def run_tool(argv: list[str], cwd: str) -> dict:
    proc = subprocess.run(
        argv,
        cwd=cwd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        check=False,
    )
    out = clip_meta(proc.stdout, CAP)
    err = clip_meta(proc.stderr, CAP)
    return {
        "span_type": "tool",
        "argv": argv,
        "cwd": str(Path(cwd).resolve()),
        "returncode": proc.returncode,
        "stdout_bytes": out["bytes"],
        "stderr_bytes": err["bytes"],
        "stdout_cap": out["cap"],
        "stderr_cap": err["cap"],
        "stdout_truncated": out["truncated"],
        "stderr_truncated": err["truncated"],
        "stdout_sha256": out["sha256"],
        "stderr_sha256": err["sha256"],
        "stdout_head_sha256": out["head_sha256"],
        "stdout_tail_sha256": out["tail_sha256"],
        "preview_stdout": out["preview"],
        "preview_stderr": err["preview"],
        "encoding": "utf-8",
        "ok_preview": proc.returncode == 0 and not out["truncated"] and not err["truncated"],
    }


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("usage: wrap_tool.py CWD CMD [ARGS...]", file=sys.stderr)
        sys.exit(2)
    record = run_tool(sys.argv[2:], sys.argv[1])
    print(json.dumps(record, ensure_ascii=False))

Example command:

python3 wrap_tool.py "$PWD" pytest -q tests/test_billing.py

Redirect that JSON line into run.jsonl.
Keep one file per agent run, not per week.
Do not append unrelated jobs into the same file.

A truncation linter

Feed traces into this checker next.
It fails when flags disagree with byte counts.
It also fails when a preview claims success after a clip.

#!/usr/bin/env python3
"""Lint tool spans for truncation contract breaks."""
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = (
    "stdout_bytes",
    "stderr_bytes",
    "stdout_cap",
    "stderr_cap",
    "stdout_truncated",
    "stderr_truncated",
)


def problems(span: dict, line_no: int) -> list[str]:
    issues: list[str] = []
    if span.get("span_type") != "tool":
        return issues
    missing = [key for key in REQUIRED if key not in span]
    if missing:
        issues.append(f"L{line_no}: missing {missing}")
        return issues
    for stream in ("stdout", "stderr"):
        n = span[f"{stream}_bytes"]
        cap = span[f"{stream}_cap"]
        flag = span[f"{stream}_truncated"]
        if cap is not None and n > cap and flag is not True:
            issues.append(f"L{line_no}: {stream} clipped without flag")
        if cap is not None and n <= cap and flag is True:
            issues.append(f"L{line_no}: {stream} flag set under cap")
        if flag and span.get(f"{stream}_sha256"):
            issues.append(f"L{line_no}: full {stream} hash on clipped stream")
    if span.get("ok_preview") is True and (
        span["stdout_truncated"] or span["stderr_truncated"]
    ):
        issues.append(f"L{line_no}: ok_preview true while truncated")
    return issues


def lint_path(path: Path) -> int:
    found = 0
    with path.open() as handle:
        for line_no, raw in enumerate(handle, start=1):
            raw = raw.strip()
            if not raw:
                continue
            span = json.loads(raw)
            for item in problems(span, line_no):
                print(item)
                found += 1
    return found


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("usage: lint_truncation.py run.jsonl", file=sys.stderr)
        sys.exit(2)
    count = lint_path(Path(sys.argv[1]))
    print(f"issues={count}")
    sys.exit(1 if count else 0)

Run it:

python3 lint_truncation.py run.jsonl

A non-zero exit means stop model debugging.
Re-run the clipped tool without a cap first.
Diff that full stderr against the stored preview.

Decision table

Observation	First action	Not the first action
`stderr_truncated=true`, returncode 0	Re-run tool with cap removed	Raise temperature
`stdout_truncated=true` on test output	Fetch tail hash, re-run pytest	Swap the coding model
both byte counts zero, returncode 0	Check pipe stall and ulimit	Add more tool descriptions
preview shows pass, linter fails	Treat span as incomplete	Close the incident
binary `encoding`, hashes only	Skip UTF-8 preview diffs	Parse preview as logs

Use the table during incident review.
Do not turn it into a model rubric.
The artifact is the span contract, not a score.

A debug loop you can reuse

Use this loop on the next opaque agent miss:

Export JSONL for that single run only.
Run lint_truncation.py run.jsonl.
Stop if any span sets a truncation flag.
Re-run that one argv with CAP removed.
Diff new stderr against preview_stderr.
Inspect prompts only after the pipe is honest.

Do not start with a new system prompt.
Do not start with a larger model.
Fix the stream, then re-evaluate the policy.

Where a scratch model and server fit

Batch linting JSONL does not need a GPU.
A small VM can parse traces and re-run tools.
That is the only capacity this method needs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Those two are enough to host the linter and cheap re-runs.
The checker does not depend on that product to work.
Remove the product and the span contract still stands.

Limitations

Hashes cannot rebuild bytes you already dropped.
Stdout may contain tokens, paths, and secrets.
Nested shells can apply a second silent cap.
Binary compilers break naive UTF-8 previews.
duration_ms still says nothing about clip point.
Preview diffs fail when logs contain timestamps.

Store redaction rules beside the collector.
Hash first if policy forbids raw command output.
Do not ship full pytest logs to a shared trace bucket.

Who should skip this

Skip this workflow if agents never call tools.
Skip it if your collector already stores full blobs.
Skip it if policy forbids retaining command output.
Skip it for interactive TTY tools with cursor codes.

Close

The model cannot repair bytes it never received.
Record truncation flags before you blame the policy.
Run lint_truncation.py on one failed JSONL file this week.

Hash the Tool Catalog Before You Trust the Trace

Riley Wang — Tue, 22 Sep 2026 14:54:48 +0000

The agent printed done after four tool calls. The JSONL file looked complete at a glance. One name in the log was run_tests.

The local harness never registered that tool name. It only exposed pytest, read_file, and grep. The runtime still accepted the invented call.

The runtime returned a one-line error string. The model apologized and called pytest next. The final assistant message still said green.

The session then burned forty minutes on prompt edits. The catalog itself never appeared in any span. That missing catalog set was the actual defect.

Why traces lie without a catalog

Most agent logs store name, arguments, and output. They skip the tool list the model received. Replay then compares calls against the wrong contract.

Three events get labeled as model failure. They are catalog events instead.

Invented names that were never in the catalog
Stale names left after a schema edit
Result rows whose tool_call_id values do not match

Happy-path tool-calling writeups skip this gap. A complete trace needs the negative path. Unknown names should be first-class events.

Record a frozen catalog at run start

Freeze the catalog before the first completion request. Sort the names for a stable order. Hash the canonical blob once.

Store these fields on a single catalog JSONL event:

run_id as a UUID string
catalog_names as a sorted list
catalog_sha256 over that name list
schema_sha256 per tool, optional but useful
started_at as a UTC timestamp

Every later span can join on catalog_sha256. Two runs with different hashes are not comparable. Diff those catalogs before you diff prompts.

Name hash versus schema hash

A name hash catches invented tools quickly. It misses field changes on the same name. Hash the JSON schema as well.

Use stable serialization for that second hash. Sort keys. Drop extra whitespace. Then SHA-256 the UTF-8 bytes.

If read_file gains an offset field, schema hash moves. The name hash stays put in that case. The debug loop can tell those two drifts apart.

Artifact: a JSONL catalog tracer

The script below is a labeled local example. It does not report any production metrics. It writes JSONL lines and a small summary table.

# catalog_trace.py
# Labeled local example. Not a measured production run.
from __future__ import annotations

import hashlib
import json
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

CATALOG = {
    "pytest": {
        "type": "object",
        "properties": {"args": {"type": "array", "items": {"type": "string"}}},
        "required": ["args"],
    },
    "read_file": {
        "type": "object",
        "properties": {"path": {"type": "string"}},
        "required": ["path"],
    },
    "grep": {
        "type": "object",
        "properties": {
            "pattern": {"type": "string"},
            "path": {"type": "string"},
        },
        "required": ["pattern", "path"],
    },
}


def utc_now() -> str:
    return datetime.now(timezone.utc).isoformat()


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def canonical_json(obj: Any) -> bytes:
    return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode("utf-8")


def catalog_event(run_id: str) -> dict[str, Any]:
    names = sorted(CATALOG)
    schemas = {
        name: sha256_bytes(canonical_json(schema))
        for name, schema in CATALOG.items()
    }
    return {
        "type": "catalog",
        "run_id": run_id,
        "started_at": utc_now(),
        "catalog_names": names,
        "catalog_sha256": sha256_bytes(canonical_json(names)),
        "schema_sha256": schemas,
    }


def call_event(
    run_id: str,
    catalog_sha: str,
    tool_call_id: str,
    name: str,
    arguments: dict[str, Any],
) -> dict[str, Any]:
    known = name in CATALOG
    flags = [] if known else ["UNKNOWN_TOOL"]
    schema_sha = None
    if known:
        schema_sha = sha256_bytes(canonical_json(CATALOG[name]))
    return {
        "type": "tool_call",
        "run_id": run_id,
        "ts": utc_now(),
        "catalog_sha256": catalog_sha,
        "tool_call_id": tool_call_id,
        "name": name,
        "known": known,
        "schema_sha256": schema_sha,
        "args_sha256": sha256_bytes(canonical_json(arguments)),
        "arguments": arguments,
        "flags": flags,
    }


def result_event(
    run_id: str,
    call_ids: set[str],
    tool_call_id: str,
    name: str,
    exit_code: int,
    output: str,
) -> dict[str, Any]:
    flags = []
    if tool_call_id not in call_ids:
        flags.append("ID_MISMATCH")
    return {
        "type": "tool_result",
        "run_id": run_id,
        "ts": utc_now(),
        "tool_call_id": tool_call_id,
        "name": name,
        "exit_code": exit_code,
        "output_bytes": len(output.encode("utf-8")),
        "output_head": output[:240],
        "flags": flags,
    }


def unused_event(run_id: str, catalog_names: list[str], used: set[str]) -> dict[str, Any]:
    unused = [name for name in catalog_names if name not in used]
    return {
        "type": "catalog_unused",
        "run_id": run_id,
        "unused_names": unused,
        "flags": ["UNUSED_TOOL"] if unused else [],
    }


def main() -> None:
    run_id = str(uuid.uuid4())
    out = Path("catalog_trace.jsonl")
    catalog = catalog_event(run_id)
    catalog_sha = catalog["catalog_sha256"]
    call_ids: set[str] = set()
    used: set[str] = set()
    rows = [catalog]

    # Invented name: not in CATALOG.
    unknown_id = "call_run_tests_1"
    rows.append(
        call_event(
            run_id,
            catalog_sha,
            unknown_id,
            "run_tests",
            {"path": "tests/"},
        )
    )
    call_ids.add(unknown_id)
    used.add("run_tests")
    rows.append(
        result_event(
            run_id,
            call_ids,
            unknown_id,
            "run_tests",
            2,
            "unknown tool: run_tests",
        )
    )

    # Registered name with a matching id.
    pytest_id = "call_pytest_1"
    rows.append(
        call_event(
            run_id,
            catalog_sha,
            pytest_id,
            "pytest",
            {"args": ["-q"]},
        )
    )
    call_ids.add(pytest_id)
    used.add("pytest")
    rows.append(
        result_event(
            run_id, call_ids, pytest_id, "pytest", 0, "2 passed in 0.40s"
        )
    )

    # Planted pairing bug: result id was never issued.
    rows.append(
        result_event(
            run_id,
            call_ids,
            "call_missing_9",
            "read_file",
            0,
            "# fake",
        )
    )

    rows.append(unused_event(run_id, catalog["catalog_names"], used))

    with out.open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, sort_keys=True) + "\n")

    print(f"run_id={run_id}")
    print(f"catalog_sha256={catalog_sha}")
    print(f"wrote {out}")
    for row in rows:
        flags = row.get("flags") or []
        if flags:
            print(f"{row['type']:16} {row.get('name', '-'):12} {flags}")


if __name__ == "__main__":
    main()

How to run the example

python3 catalog_trace.py
wc -l catalog_trace.jsonl

The demo should print three flag groups. Treat them as fixtures, not live traffic.

UNKNOWN_TOOL on the run_tests call
ID_MISMATCH on the planted result row
UNUSED_TOOL for grep and read_file

Query the JSONL without a UI

python3 - <<'PY'
import json
from pathlib import Path

rows = [json.loads(line) for line in Path("catalog_trace.jsonl").read_text().splitlines()]
catalog = next(r for r in rows if r["type"] == "catalog")
print("catalog", catalog["catalog_sha256"], catalog["catalog_names"])

for row in rows:
    if row.get("flags"):
        print(row["type"], row.get("name"), row["flags"])

calls = {r["tool_call_id"]: r for r in rows if r["type"] == "tool_call"}
results = [r for r in rows if r["type"] == "tool_result"]
for result in results:
    matched = result["tool_call_id"] in calls
    print("paired" if matched else "orphan", result["tool_call_id"], result["name"])
PY

Optional jq checks if you already use it:

jq -r 'select(.type=="catalog") | .catalog_sha256' catalog_trace.jsonl
jq -c 'select(.flags != null and (.flags|length)>0)' catalog_trace.jsonl

A reusable debug loop

Do not start with the model card. Start with catalog identity. Then inspect pairing.

Write one catalog event before any completion.
Flag or reject names outside that frozen set.
Copy tool_call_id from each call into its result.
Flag ID_MISMATCH when those identifiers diverge.
Emit unused catalog names after the last span.
Compare two runs only after both hashes match.

Decision table

Symptom	First check	If catalog hash differs	If catalog hash matches
Invented name in the log	Membership in `catalog_names`	Wrong catalog shipped	Model invented a tool
Same name, rejected arguments	Per-tool `schema_sha256`	Tool definition drift	Argument validation bug
Result row without a call	`tool_call_id` index	Mixed JSONL files	Runtime pairing bug
Green summary after a failed tool	Unknown plus unused flags	Recovery hid a catalog miss	Prompt papered over the error

Keep argument hashes, not raw secrets. Redact tokens before you write JSONL. Hash the redacted object, not the live credential.

Parallel tool calls still need unique ids. Two read_file spans can share a name. They cannot share a tool_call_id.

Empty catalogs are events too. A zero-length name list should still hash. A later call against that hash is always UNKNOWN_TOOL.

Keep the tracer on the same host

Traces rot when they leave the machine that ran tools. Hash the catalog on that host. Store JSONL beside the prompt file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. That pairing is enough to keep catalog events, tool spans, and prompts on one box.

If you already schedule runs there, place catalog_trace.py next to the harness. The loop does not need a separate cluster.

What this loop does not catch

A catalog hash does not prove the tool body is correct. A registered pytest call can still run the wrong tests. You still need exit codes and file diffs for that class.

Name membership does not validate argument types at runtime. Add JSON Schema checks if your harness can afford them. Schema hashes only detect definition drift.

This tracer does not reconstruct truncated model streams. It records parsed objects after JSON decode. Mid-stream parse failures need a raw argument buffer.

It also does not fix clock skew across hosts. One host, one JSONL file, one clock. Split files across machines without a run id and pairing collapses.

Do not treat unused tools as proof of a bad prompt. Some catalogs include rare escape hatches. Mark those names as optional if that is the contract.

Who should skip this approach

Skip this loop if the agent has no tools. A chat-only completion has no catalog to hash. Log finish reasons instead.

Skip it if your runtime already emits OpenTelemetry spans. Those spans must already include catalog hash and call ids. Duplicating JSONL then adds noise.

Skip it for high-volume production without sampling. Full argument blobs will dominate disk. Keep hashes and flags, drop payloads.

Skip it when several agents share one file without a role field. Mix two catalogs in one JSONL and membership checks lie. Give each role its own run_id.

Close the loop in this order

Unknown tool names are trace events. They are not mysterious weight drift. Hash the catalog first. Then debug the model.

Match tool_call_id on every result row. List unused names at the end. Only then compare two runs that share a catalog hash.

The Timeline Has Holes. Record Span Sequence Numbers.

Riley Wang — Mon, 21 Sep 2026 13:02:49 +0000

The on-call channel lit up after a staging deploy. The agent report claimed two full pytest runs. The trace view showed four tool spans only. Two pytest spans never reached the collector backend.

This is a reconstructed incident, not a dump. The failure mode is omission, not a wrong token.

Green summaries hide missing spans

Teams debug the prompt when the timeline is incomplete. A missing tool span looks like a skipped step. Operators then change temperature, tools, or system text.

The model may have called the tool correctly. The exporter dropped the span during a batch flush. You cannot tell without a per-run sequence number.

Dashboards also reward short, pretty traces. Sparse timelines look efficient during a review. They can hide work that never left the process.

Sequence is not a timestamp

Clock order was a different failure class. Sequence answers a narrower question than time. Did this run emit span k, yes or no.

Timestamps still matter for latency later. They do not prove the exporter kept every span. A hole can sit between two plausible timestamps.

The contract for one run

One agent run needs one strictly increasing integer. Start at one when the run record is created. Increment after each tool call, model step, and flush.

Store seq before the exporter touches the row. Never renumber after a retry or partial export. Export retries keep the original number on that span.

A real tool retry is a new span. It gets the next integer, not a reused one. Mixing those two retry kinds creates false duplicates.

Fields to persist

Keep the row small and boring:

run_id: one id for the whole attempt
seq: integer starting at one, no intended gaps
span_id: unique id for this span row
name: tool name or model step label
export_batch_id: flush batch that held the row
t_emit_ms: monotonic clock at emit time
attempt: retry index for that tool name

Reject rows that lack run_id or seq. Incomplete telemetry is a failed run, not a maybe.

Stamp seq in a thin wrapper

Put the counter in process memory, not the model. The wrapper owns seq; the model never sees it. The example below is a local script, not production telemetry.

# span_seq.py — example wrapper, run locally
from __future__ import annotations

import json
import time
import uuid
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable


@dataclass
class SeqTracer:
    path: Path
    run_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
    _seq: int = 0
    _batch: int = 0

    def emit(self, name: str, attempt: int = 0, **fields: Any) -> dict:
        self._seq += 1
        row = {
            "run_id": self.run_id,
            "seq": self._seq,
            "span_id": uuid.uuid4().hex[:16],
            "name": name,
            "attempt": attempt,
            "export_batch_id": f"b{self._batch}",
            "t_emit_ms": time.monotonic_ns() // 1_000_000,
            **fields,
        }
        self.path.parent.mkdir(parents=True, exist_ok=True)
        with self.path.open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(row, ensure_ascii=True) + "\n")
        return row

    def flush_batch(self) -> None:
        self._batch += 1


def wrap_tool(tracer: SeqTracer, name: str, fn: Callable[..., Any]) -> Callable[..., Any]:
    def wrapped(*args: Any, **kwargs: Any) -> Any:
        attempt = int(kwargs.pop("_attempt", 0))
        tracer.emit(name, attempt=attempt, phase="start")
        try:
            result = fn(*args, **kwargs)
        except Exception as exc:
            tracer.emit(name, attempt=attempt, phase="error", error=type(exc).__name__)
            raise
        tracer.emit(name, attempt=attempt, phase="end")
        return result

    return wrapped

Flush after a successful export ACK, not before write. A killed process then leaves a tail gap. That gap is data, not a model mystery.

Fail the debug session on holes

Do not open the prompt until this script exits zero. The checker reads JSONL and groups rows by run_id. It expects seq values equal to one through N.

Any hole, duplicate, or restart fails the run. Save the file and run it against one trace.

# check_span_seq.py — example completeness checker
from __future__ import annotations

import argparse
import json
import sys
from collections import defaultdict
from pathlib import Path


def load_rows(path: Path) -> list[dict]:
    rows = []
    with path.open(encoding="utf-8") as fh:
        for line_no, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                row = json.loads(line)
            except json.JSONDecodeError as exc:
                raise SystemExit(f"bad json at line {line_no}: {exc}") from exc
            if "run_id" not in row or "seq" not in row:
                raise SystemExit(f"missing run_id/seq at line {line_no}")
            rows.append(row)
    return rows


def report_for_run(run_id: str, rows: list[dict]) -> dict:
    seqs = [int(r["seq"]) for r in rows]
    names = {int(r["seq"]): r.get("name", "?") for r in rows}
    last = max(seqs) if seqs else 0
    present = set(seqs)
    expected = set(range(1, last + 1)) if last else set()
    missing = sorted(expected - present)
    dupes = sorted(s for s in present if seqs.count(s) > 1)
    completeness = (len(present) / last) if last else 0.0
    return {
        "run_id": run_id,
        "row_count": len(rows),
        "last_seq": last,
        "missing": missing,
        "duplicates": dupes,
        "first_gap_name_after": names.get((missing[0] - 1), None) if missing else None,
        "completeness": round(completeness, 4),
        "ok": not missing and not dupes and last == len(present),
    }


def main() -> int:
    p = argparse.ArgumentParser(description="Detect span seq gaps in JSONL")
    p.add_argument("jsonl", type=Path)
    p.add_argument("--run-id", default="")
    args = p.parse_args()
    grouped: dict[str, list[dict]] = defaultdict(list)
    for row in load_rows(args.jsonl):
        grouped[str(row["run_id"])].append(row)
    run_ids = [args.run_id] if args.run_id else sorted(grouped)
    failed = 0
    for run_id in run_ids:
        rep = report_for_run(run_id, grouped.get(run_id, []))
        print(json.dumps(rep, indent=2))
        failed += 0 if rep["ok"] else 1
    return 2 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

python3 check_span_seq.py traces/staging-0214.jsonl --run-id staging-0214
echo $?

Exit zero means the sequence is dense. Exit two means you still lack a timeline. Treat that like a red test, not a hint.

Read the gap shape, not the model card

The exit code is the first signal, not the model. Completeness is present integers divided by last_seq. The shape of the hole picks the next probe.

Gap shape	Likely cause	Next probe
`1,2,4,5`	middle drop or flush race	`export_batch_id` on neighbors of 3
`1,2,3` then stop	tail drop or kill	last flush, process exit code
`1,1,2`	duplicate export	ACK path versus in-memory buffer
`5,6,7`	missing head	late attach, wrong `run_id`
two dense streams	mixed runs	join keys, child trace ids

A middle hole after tool.pytest start is not a skip. It is often a dropped end span. The agent may have finished the tests off-screen.

A tail hole after model.generate often means SIGKILL. Disk full and batch limits look the same here. Inspect the collector log before the weights.

A four-step debug loop

Use the same order on every flaky agent:

Confirm one run_id and a dense seq range.
Only then compare tool names against the agent summary.
Diff payloads after the timeline is complete.
Change prompts last, never as the first move.

Skip step one and you debug fiction. The summary text is not an inventory. It is another span you might also have dropped.

Reproducible fixture

Label this fixture as an unexecuted example unless you save it. Three rows should fail with missing: [3].

{"run_id":"staging-0214","seq":1,"name":"tool.read","export_batch_id":"b0"}
{"run_id":"staging-0214","seq":2,"name":"tool.pytest","export_batch_id":"b0"}
{"run_id":"staging-0214","seq":4,"name":"model.generate","export_batch_id":"b1"}

Expected report fields for that file:

last_seq: 4
missing: [3]
completeness: 0.75
process exit: 2

If your checker prints ok: true here, stop. The tool is wrong, not the agent.

Where a throwaway box fits

You can run the wrapper beside a local collector. A laptop works until the disk fills with JSONL. Overnight reproductions need a box that stays up.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use the free server to host the collector and wrapper. Use free model access only to reproduce the same traces.

Do not treat that access as a quota, SLA, or model list. If you need a disposable host, try the free server option. Keep the JSONL on that host until completeness stays at one.

Limitations

Sequence numbers do not prove the tool did useful work. They only prove the tracer emitted a row. Semantic bugs still need diffs and exit codes elsewhere.

In-process counters die with the process. A crash before emit looks like a skip. You still need an OS-level audit for that class.

Concurrent workers need a lock or a partition key. Two threads sharing one integer will duplicate seq. This article does not cover distributed id factories.

Sampling in a real collector can drop spans after emit. Your JSONL can be complete while the vendor UI is not. Compare both stores before you trust either.

Who should not use this

Skip this loop if OpenTelemetry already ACKs every span. Skip it if you lack a single run_id today. Skip it for vibe checks and chat demos with no tools.

Do not use it as a model leaderboard. Completeness is not quality. A dense trace can still call the wrong binary.

Platform teams with an existing trace pipeline should extend that pipeline. Do not add a second JSONL dialect without a join key. Two incomplete stores are worse than one.

Tomorrow morning

Pick one flaky staging run, not a new model. Add seq on the wrapper before the exporter. Fail the debug session when completeness is below one.

Only then inspect prompts, tools, and model output. The timeline had holes. Number every span until it does not.

The Agent Reported Green. Record argv and the Exit Code.

Riley Wang — Sun, 20 Sep 2026 11:12:50 +0000

The agent posted a green summary after three tool calls.
The pull request still failed the same pytest file.
The trace listed a test span lasting 40 milliseconds.
No argv was stored, and no exit code was stored.

That gap is the real bug in the loop.
The model text is not a process witness.

The failure mode

Agents summarize work they did not run.
A span named run_tests is not evidence.
A chat reply that says tests passed is not evidence.
The operating system watched the child process. Your recorder should watch it too.

Three fields close most of the gap:

argv as a JSON array, never a slogan
exit code from a real wait
wall time in milliseconds

Add two hashes when output can still lie:

sha256 of stdout bytes
sha256 of stderr bytes

Do not hash the git tree here.
That is a different invariant, already covered elsewhere.
This loop asks a narrower question: did a process run?

Why duration is a cheap invariant

A pytest run on a real tree takes seconds.
A 40 millisecond span did not collect tests.
It skipped, cached, or never launched.

Treat duration as a floor, not a trophy.
Compare it to a limit you set per command class.
Measure your own repo once. Then store the floor beside the prefix.

Proposal floors, not measured claims:

command class	min wall ms	why it exists
`pytest`	500	import and collect
`python -m pytest`	500	same runner, different argv
`npm test`	800	process boot
`make`	200	at least one recipe
`git`	20	cheap, allow short

Do not copy these numbers into production blindly.
A tiny fixture suite can beat 500 milliseconds honestly.
A cached runner can also beat the floor honestly.
When that happens, lower the floor or pin the cache flag in argv.

A labeled example

The following run is a constructed example.
It is not a production postmortem.

The agent wrote: "I ran pytest and everything passed."
The span duration was 38 milliseconds.
The recorded argv was ["echo", "passed"].
The exit code was 0. The stdout hash matched echo.

The summary was true as prose.
The work never happened.
CI should fail that run before anyone debugs the model.

Schema for one process span

Store one JSON object per child process.
Append it to a JSONL file beside the agent trace.
Keep the file outside the repo under test.

{
  "span_id": "proc-014",
  "parent_tool_span": "tool-003",
  "argv": ["python", "-m", "pytest", "tests/test_checkout.py", "-q"],
  "cwd": "/work/repo",
  "exit_code": 1,
  "duration_ms": 1842,
  "stdout_sha256": "a3c1b0e4f8d2c91a7b55e0c1d4a8f3269b0e1c2d3a4b5c6d7e8f9012ab34cd56",
  "stderr_sha256": "90bb12aa44cc78de9012ff3456ab78cd9012ee34aa56bb78cc90dd12ef34aa56",
  "stdout_bytes": 612,
  "stderr_bytes": 88,
  "env_names": ["PATH", "PYTHONPATH"],
  "duration_floor_ms": 500,
  "duration_ok": true
}

Use full hex in the real file.
Never collapse argv to the string pytest.
pytest tests/ is not pytest tests/test_checkout.py.
The selector is part of the evidence.

Skip environment values. Record env names only.
Secrets leak through traces faster than through application code.

Recorder you can attach to the tool handler

Wrap every tool that claims to run a command.
Do not parse the model's summary after the fact.
Proposal: a small Python helper. Label it unexecuted until you run it.

import hashlib
import json
import os
import subprocess
import time
from pathlib import Path

TRACE = Path(os.environ.get("PROCESS_TRACE", "process_spans.jsonl"))

FLOORS = {
    "pytest": 500,
    "python -m pytest": 500,
    "npm test": 800,
    "make": 200,
    "git": 20,
}

DENY_BINARIES = {"echo", "true", "false", "cat", "printf"}

def floor_for(argv):
    joined = " ".join(argv)
    for prefix, ms in FLOORS.items():
        if joined.startswith(prefix):
            return ms
    return 0

def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def run_recorded(argv, cwd=None, span_id=None, parent=None):
    if not argv:
        raise ValueError("argv is required")
    cwd = cwd or os.getcwd()
    t0 = time.monotonic()
    proc = subprocess.run(
        argv,
        cwd=cwd,
        capture_output=True,
        check=False,
    )
    duration_ms = int((time.monotonic() - t0) * 1000)
    floor = floor_for(argv)
    rec = {
        "span_id": span_id,
        "parent_tool_span": parent,
        "argv": argv,
        "cwd": str(Path(cwd).resolve()),
        "exit_code": proc.returncode,
        "duration_ms": duration_ms,
        "stdout_sha256": sha256_bytes(proc.stdout),
        "stderr_sha256": sha256_bytes(proc.stderr),
        "stdout_bytes": len(proc.stdout),
        "stderr_bytes": len(proc.stderr),
        "env_names": sorted(os.environ.keys()),
        "duration_floor_ms": floor,
        "duration_ok": duration_ms >= floor,
        "denied_binary": argv[0] in DENY_BINARIES,
    }
    with TRACE.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec) + "\n")
    return proc

Call run_recorded from the tool handler.
Do not let the model shell out around this wrapper.
If the agent can reach raw bash -c, the JSONL file will stay quiet.
A quiet file is a signal. Treat silence as a failed exec.

Debug loop after a green reply

Use the JSONL file before you debug the model.
Do not open token logs first.

Filter spans whose argv looks like a test runner.
Reject spans with missing argv or missing exit code.
Reject spans where duration_ok is false.
Reject spans where denied_binary is true.
Diff stdout hashes across retries of the same argv.
Only then read the model tokens.

First-pass filter:

python -c "
import json
from pathlib import Path
need = {'pytest', 'npm', 'make', 'go', 'cargo'}
for line in Path('process_spans.jsonl').read_text().splitlines():
    rec = json.loads(line)
    argv = rec.get('argv') or []
    if need.intersection(argv) or any(need.intersection(a.split()) for a in argv):
        print(json.dumps(rec, indent=2))
"

Stricter check with jq:

jq -c 'select(.exit_code != 0 or .duration_ok == false or .denied_binary == true)' process_spans.jsonl

If the file is empty, the agent never launched tests.
The green reply is a story. Stop there.
If jq prints rows, debug those rows before the prompt.

Decision table

observation	likely cause	next action
argv missing	tool did not exec	fix the wrapper
argv is echo/true/cat	skipped work	fail the run
exit 0, duration below floor	fake or cache	fail, then retune floor
exit 1, hash stable	real failing test	debug the test
exit 0, hash changes	flaky output	pin flags in argv
cwd outside the repo	wrong tree	abort
JSONL empty after a green reply	wrapper bypass	block merge

Keep the table next to the recorder.
Update floors when the suite grows.
Do not turn floors into user-facing SLOs.

Correlate with the parent tool span

Every process record needs parent_tool_span.
Without it, you cannot join command facts to the agent trace.
A test process with no parent is an orphan. Drop the run.

Join rule, labeled as a proposal:

Load agent spans from the existing JSONL trace.
Load process spans from process_spans.jsonl.
Index process rows by parent_tool_span.
For each tool span named like run_tests, require one child process.
Fail if the child argv does not match the tool arguments.

Mismatch is common.
The tool payload says pytest tests/test_checkout.py.
The process argv says pytest -q.
That is a different test set. Record both. Do not merge them.

CI gate

Put a gate in CI that never reads model text.
It only reads process records.

# check_process_trace.py — proposal, run against committed fixtures first
import json
import sys
from pathlib import Path

path = Path(sys.argv[1])
if not path.exists() or path.stat().st_size == 0:
    raise SystemExit("no process spans; agent did not exec")

errors = []
for line in path.read_text().splitlines():
    rec = json.loads(line)
    if rec.get("denied_binary"):
        errors.append(f"denied argv {rec.get('argv')}")
    if rec.get("exit_code") not in (0,):
        errors.append(f"exit {rec.get('exit_code')} argv={rec.get('argv')}")
    if rec.get("duration_ok") is False:
        errors.append(f"slow-path skipped argv={rec.get('argv')}")
    if not rec.get("argv"):
        errors.append("missing argv")

if errors:
    print("\n".join(errors))
    raise SystemExit(1)
print(f"ok {sum(1 for _ in path.open())} process spans")

Run it after the agent job, not before.
An empty file must fail. A green chat must not pass this script.

What this does not prove

A valid argv does not prove coverage.
Exit code 0 does not prove the right tests ran.
A duration above the floor does not prove correctness.
Hashes still match when output is truncated nonsense.

Also record the test selector in argv.
Clock skew can still disorder parent spans.
This article does not replace a trace clock check.
It also does not replace a working-tree hash after writes.

Redact argv tokens that look like secrets.
Flags such as --token do not belong in JSONL.
Replace the value with *** before append.
Keep the flag name so you can still see the shape.

Where a free runner helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The recorder needs a host that actually execs commands.
A notebook that only echoes model text cannot prove work.
MonkeyCode's free model access and free server option can host this wrapper beside the agent process.
The method still works on any box you control.
If you already need a remote runner, keep this JSONL next to the trace.

Limitations

Windows argv quoting differs from POSIX lists.
Some tools daemonize and return early.
Cached test runners can beat your floor honestly.
Hashing huge stdout can stall the event loop.
Env name lists can still reveal stack shape.
bash -c collapses argv into one opaque string.
Floors drift as fixtures grow or shrink.

Cap hashed bodies by byte length if needed.
Hash the first N bytes plus the total length.
That still detects empty fakes. It will miss late differences.

Who should not use this

Do not add this wrapper on untrusted multi-tenant hosts.
Do not store stdout bodies when they may hold secrets.
Do not treat floors as product benchmarks.
Do not use this as a model leaderboard.

If your agent never shells out, skip this pattern.
Log the tool payload instead.
If you only need git state, hash the tree, not stdout.

Checklist before you merge the recorder

JSONL path sits outside the repo under test
Secrets are stripped from argv before append
Floors come from one local measurement
jq or check_process_trace.py runs in CI
Failed duration or exit blocks the merge
Parent span ids join to the agent trace
Empty process files fail the job

Green text is cheap. Process records are not.
Record argv. Record the exit code. Then debug.

Lint Agent Traces Before You Debug the Model

Riley Wang — Fri, 18 Sep 2026 07:26:46 +0000

A Monday CI job went green after the agent ran. The pull request looked small and well scoped. A later job failed on a path the agent touched.

You opened the JSONL trace to reconstruct the tool calls. Span four named a formatter and nothing else. There was no argument digest and no exit code.

You started rewriting the system prompt that morning. The model was not the missing piece here. The tracer had emitted a success-shaped hole in the log.

Incomplete traces impersonate engineering

Hot threads this week treat vibe coding as a taste problem. The sharper failure is missing evidence. A green agent status is not a measurement.

A usable trace is a closed record. Every span kind needs a required field set. If a field is absent, the run is not debuggable.

Stop reading the model output first. Lint the file. Fix the tracer. Only then inspect behavior.

A closed attribute set

Keep four span kinds. Do not invent extra kinds on day one. Reject unknown kinds in CI.

span_kind	required fields
`run.start`	`trace_id`, `span_id`, `ts_utc`, `task_id`, `repo_head`
`model.completion`	`trace_id`, `span_id`, `parent_id`, `prompt_digest`, `response_digest`, `schema_digest`, `duration_ms`
`tool.call`	`trace_id`, `span_id`, `parent_id`, `tool_name`, `schema_digest`, `args_digest`, `args_json_valid`, `exit_code`, `duration_ms`, `stdout_digest`, `stderr_digest`, `result_kind`
`run.end`	`trace_id`, `span_id`, `parent_id`, `stop_class`, `ok`

schema_digest is the hash of the tool schema the model saw. args_json_valid is a boolean from local JSON Schema checks. result_kind must be ok, exception, timeout, or rejected.

unknown is not allowed. Missing keys are not allowed. Nulls are not allowed for required fields.

Artifact: lint JSONL before anyone reads it

Label: this is a local example, not a production study. Save it as lint_agent_trace.py. Run it on one file.

#!/usr/bin/env python3
"""Fail a trace file when required span fields are missing."""
from __future__ import annotations

import hashlib
import json
import sys
from typing import Any

REQUIRED = {
    "run.start": [
        "trace_id", "span_id", "ts_utc", "task_id", "repo_head"
    ],
    "model.completion": [
        "trace_id", "span_id", "parent_id", "prompt_digest",
        "response_digest", "schema_digest", "duration_ms",
    ],
    "tool.call": [
        "trace_id", "span_id", "parent_id", "tool_name",
        "schema_digest", "args_digest", "args_json_valid",
        "exit_code", "duration_ms", "stdout_digest",
        "stderr_digest", "result_kind",
    ],
    "run.end": ["trace_id", "span_id", "parent_id", "stop_class", "ok"],
}

ALLOWED_RESULT = {"ok", "exception", "timeout", "rejected"}
ALLOWED_STOP = {"completed", "budget", "error", "cancelled"}


def digest(value: Any) -> str:
    blob = json.dumps(value, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:16]


def lint_line(n: int, row: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    kind = row.get("span_kind")
    if kind not in REQUIRED:
        return [f"L{n}: unknown span_kind {kind!r}"]
    for key in REQUIRED[kind]:
        if key not in row or row[key] in (None, ""):
            errors.append(f"L{n}: {kind} missing {key}")
    if kind == "tool.call":
        if row.get("args_json_valid") not in (True, False):
            errors.append(f"L{n}: args_json_valid must be bool")
        if row.get("result_kind") not in ALLOWED_RESULT:
            errors.append(f"L{n}: bad result_kind {row.get('result_kind')!r}")
        if not isinstance(row.get("exit_code"), int):
            errors.append(f"L{n}: exit_code must be int")
    if kind == "run.end" and row.get("stop_class") not in ALLOWED_STOP:
        errors.append(f"L{n}: bad stop_class {row.get('stop_class')!r}")
    return errors


def lint_file(path: str) -> int:
    errors: list[str] = []
    kinds: list[str] = []
    trace_ids: set[str] = set()
    with open(path, encoding="utf-8") as handle:
        for n, line in enumerate(handle, 1):
            line = line.strip()
            if not line:
                continue
            row = json.loads(line)
            kinds.append(row.get("span_kind"))
            if "trace_id" in row:
                trace_ids.add(row["trace_id"])
            errors.extend(lint_line(n, row))
    if kinds[:1] != ["run.start"] or kinds[-1:] != ["run.end"]:
        errors.append("file must start with run.start and end with run.end")
    if len(trace_ids) != 1:
        errors.append(f"expected one trace_id, found {sorted(trace_ids)}")
    for item in errors:
        print(item)
    print(f"spans={len(kinds)} errors={len(errors)}")
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(lint_file(sys.argv[1]))

A failing fixture looks like this. Note the empty tool span.

{"span_kind":"run.start","trace_id":"t1","span_id":"s0","ts_utc":"2026-09-18T12:00:00Z","task_id":"fix-null","repo_head":"abc1234"}
{"span_kind":"tool.call","trace_id":"t1","span_id":"s1","parent_id":"s0","tool_name":"format"}
{"span_kind":"run.end","trace_id":"t1","span_id":"s2","parent_id":"s0","stop_class":"completed","ok":true}

Run the linter. It should exit non-zero.

python3 lint_agent_trace.py bad.jsonl
# L2: tool.call missing schema_digest
# L2: tool.call missing args_digest
# L2: tool.call missing args_json_valid
# L2: tool.call missing exit_code
# L2: tool.call missing duration_ms
# L2: tool.call missing stdout_digest
# L2: tool.call missing stderr_digest
# L2: tool.call missing result_kind
# spans=3 errors=8

A passing tool span carries digests, not raw dumps. Store payloads elsewhere under those keys.

{"span_kind":"tool.call","trace_id":"t1","span_id":"s1","parent_id":"s0","tool_name":"format","schema_digest":"9c1e0b4a2d77f1c0","args_digest":"e3b0c44298fc1c14","args_json_valid":true,"exit_code":0,"duration_ms":41,"stdout_digest":"2c26b46b68ffc68f","stderr_digest":"e3b0c44298fc1c14","result_kind":"ok"}

Wire it as a gate. Do not review a trace that fails this command.

python3 lint_agent_trace.py run.jsonl && echo TRACE_OK

A debug loop that refuses partial evidence

Use the same loop on every incident. Do not skip step one because the reply looks fluent.

Export one JSONL file per agent run.
Lint it. If lint fails, patch the tracer.
Confirm schema_digest is stable for the same tool version.
Confirm args_json_valid is true before you blame the model.
Compare prompt_digest across reproductions of the same task.
Only then read payloads behind the digests.

Step three catches schema drift. The model may call format after the argument shape changed. The name matches. The digest does not.

Step four catches parse theater. The wrapper swallowed a JSON error. The span still said success. The linter now forces a boolean.

Step five tells you whether the prompt moved. If digests differ, you are not comparing the same request.

Cheap reproductions need a complete tracer first

You need volume to learn which fields drop. Local laptops hide that. A shared box makes the gap obvious.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host the same linted runner without a private GPU queue. That is the only product claim used here.

Keep the server as a recorder. Do not treat it as a judge. Copy lint_agent_trace.py onto the box. Drop incomplete files. Diff the survivors.

mkdir -p traces/ok traces/drop
for f in traces/inbox/*.jsonl; do
  if python3 lint_agent_trace.py "$f"; then
    mv "$f" traces/ok/
  else
    mv "$f" traces/drop/
  fi
done
python3 - <<'PY'
import json, pathlib
from collections import Counter
c = Counter()
for path in pathlib.Path("traces/drop").glob("*.jsonl"):
    for line in path.read_text().splitlines():
        if not line.strip():
            continue
        row = json.loads(line)
        if row.get("span_kind") == "tool.call" and "schema_digest" not in row:
            c[row.get("tool_name","?")] += 1
print(c.most_common(10))
PY

The histogram names wrappers that emit hollow spans. That list is your sprint board. It is not a model ranking.

What the linter cannot claim

A lint-clean trace can still be wrong. Complete fields are not correct fields. Digests do not prove a patch fixed the bug.

This workflow also fails open if you store secrets in payloads. Hash first. Redact first. Then archive. The linter does not replace access control.

Do not use this approach for single-shot chat with no tools. The required set assumes tool spans exist. Do not use it as a model leaderboard. It measures record quality, not answer quality.

Skip it if your org cannot export traces off a laptop. Skip it if you still log with unstructured print statements only. Adopt a schema first.

Practical bounds

Keep duration_ms as an integer millisecond count. Do not mix clocks in one field. Keep one trace_id per file. Split retries into new files if wrappers restart.

Store raw stdout off-trace. The lint file should stay small. Large traces hide the missing key you needed.

Re-run the linter in CI on fixture files. A tracer change that drops args_json_valid should fail the build. That is the whole point.

Close the hole, then argue about the model

Prompt edits are cheap and noisy. Tracer edits are boring and durable. If span four has no exit code, you cannot replay it.

Lint the JSONL. Count missing keys. Repair the emitter. After the file is complete, the model discussion has evidence.

If you want a shared box for those lint-clean reproductions, MonkeyCode's free model access and free server option are enough to start collecting files.

After Each Tool Span, Hash the Working Tree

Riley Wang — Thu, 17 Sep 2026 05:33:59 +0000

A coding agent prints Task complete at 02:14.
The trace shows one successful apply_patch span.
The unit tests still fail on the next command.

The log looks honest. The tree is not.
The span recorded a tool name and a short result string.
It never recorded the bytes that landed on disk.

This gap is now common in agent debugging.
Model quality debates miss it.
The missing join sits between the tool span and the working tree.

The failure is a join, not a vibe

Public threads still argue whether models already outcode most developers.
That debate does not help a failing patch land.
You need a join key from each tool span to the files it touched.

A green span only means the wrapper returned.
It does not mean the file changed.
It does not mean the change matches the argument digest.

Treat three records as one debug unit:

the tool span id and name
a stable hash of the tool arguments
a hash of selected paths after the call

If any piece is missing, stop reading the dashboard.
Start reading the join file.

A scene you can reproduce

Create a tiny repo. One module. One test. One agent tool.
The tool claims it patched tax.py.
The test still expects the old rounding path.

workspace/
  tax.py
  test_tax.py
  traces/joins.jsonl

tax.py before the run:

def round_cents(amount: float) -> int:
    return int(amount * 100)

test_tax.py:

from tax import round_cents


def test_round_cents_bankers() -> None:
    assert round_cents(1.225) == 122

The agent wrapper returns {"ok": true, "path": "tax.py"}.
Pytest still fails.
Without a tree hash, you debug the model.
With a tree hash, you debug the write.

Artifact: span-to-tree join records

Label the next block as a local helper, not a production SDK.
It is a proposal you can run in a scratch repo.
It writes one JSONL line after every tool call.

# join_trace.py — example helper, run locally
from __future__ import annotations

import hashlib
import json
import time
from pathlib import Path
from typing import Callable, Iterable

JOIN_PATH = Path("traces/joins.jsonl")
WATCH = ("tax.py", "test_tax.py")


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def hash_args(payload: object) -> str:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return sha256_bytes(blob.encode("utf-8"))


def hash_tree(root: Path, rel_paths: Iterable[str]) -> dict[str, str]:
    out: dict[str, str] = {}
    for rel in rel_paths:
        path = root / rel
        if not path.is_file():
            out[rel] = "MISSING"
            continue
        out[rel] = sha256_bytes(path.read_bytes())
    return out


def append_join(record: dict) -> None:
    JOIN_PATH.parent.mkdir(parents=True, exist_ok=True)
    with JOIN_PATH.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")


def call_with_join(
    root: Path,
    span_id: str,
    tool_name: str,
    args: dict,
    tool: Callable[[dict], dict],
) -> dict:
    before = hash_tree(root, WATCH)
    started = time.time_ns()
    result = tool(args)
    ended = time.time_ns()
    after = hash_tree(root, WATCH)
    changed = sorted(path for path in WATCH if before.get(path) != after.get(path))
    append_join(
        {
            "span_id": span_id,
            "tool_name": tool_name,
            "args_sha256": hash_args(args),
            "result_sha256": hash_args(result),
            "before": before,
            "after": after,
            "changed_paths": changed,
            "started_ns": started,
            "ended_ns": ended,
        }
    )
    return result

The helper does not need a vendor SDK.
It needs a root path, a watch list, and one append.
Keep the watch list small. Hashing the whole monorepo is waste.

A fake tool that lies without writing

This example is labeled and unexecuted in this article.
It shows a wrapper that returns success and writes nothing.

# example_only.py — labeled example, not a live agent
from pathlib import Path

from join_trace import call_with_join


def apply_patch(args: dict) -> dict:
    # Looks complete. Does not touch tax.py.
    return {"ok": True, "path": args["path"], "bytes": 214}


if __name__ == "__main__":
    root = Path(".")
    call_with_join(
        root,
        span_id="span-01",
        tool_name="apply_patch",
        args={"path": "tax.py", "patch": "use bankers rounding"},
        tool=apply_patch,
    )

Run it, then read the JSONL line.
changed_paths will be empty.
after["tax.py"] will match before["tax.py"].
The span can still say ok.

That is the bug class.
The model may have planned a patch.
The tree never moved.

The debug loop

Do not start with prompt edits.
Start with a four-step join check.

Load the latest join line for the failing span.
Confirm args_sha256 matches the prompt’s tool payload.
Confirm changed_paths is not empty when a write was claimed.
Confirm after hashes match a second hash taken by hand.

# check_join.py — example checker
import json
from pathlib import Path

from join_trace import WATCH, hash_tree

line = Path("traces/joins.jsonl").read_text(encoding="utf-8").splitlines()[-1]
rec = json.loads(line)
live = hash_tree(Path("."), WATCH)

mismatches = [p for p in WATCH if rec["after"].get(p) != live.get(p)]
claimed_write = rec["tool_name"] in {"apply_patch", "write_file"}
no_change = rec["changed_paths"] == []

print("span_id", rec["span_id"])
print("claimed_write", claimed_write)
print("changed_paths", rec["changed_paths"])
print("after_vs_live", mismatches)
print("lie_score", int(claimed_write and no_change))

Interpret the checker with a small table.
Do not treat the table as a benchmark.
It is a decision aid for one repo.

claimed write	changed_paths	after vs live	next action
yes	empty	match	wrapper never wrote; fix the tool
yes	nonempty	mismatch	later process mutated the tree
yes	nonempty	match	tree moved; debug the patch content
no	nonempty	match	unexpected mutation; find the extra writer
no	empty	match	no fs side effect; look at HTTP or memory

Most “the model is dumb” nights fall in row one.
The span was green.
The file digest never changed.

Commands for a clean replay

Keep the commands boring. Boring commands survive on-call.

git status --short
python -m pytest test_tax.py -q
python example_only.py
python check_join.py
sha256sum tax.py test_tax.py

If git status is dirty and changed_paths is empty, your tracer is blind.
If both are clean and tests fail, the patch never targeted the assertion.
If after and sha256sum disagree, something wrote after the span closed.

Record that third writer as its own span.
Formatters, codegen, and test fixtures belong in the join file.
Hidden writers make agent diffs look random.

Where a free remote loop helps

Local joins catch silent writes.
Remote joins catch “works on my laptop” trees.
Same JSONL schema. Different disk.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option.
Those two facts are the only product claims used here.
No model names, quotas, or hardware details are assumed.

A useful remote setup is narrow.
Copy the watch list, the helper, and the failing test.
Run the same tool wrapper against the same files.
Compare args_sha256 and after hashes across hosts.

If the hashes match and tests still disagree, the environment is the bug.
If the hashes differ, the write path is the bug.
Either result is cheaper than another prompt rewrite.

Use the remote box as a second witness, not as a demo.
Keep secrets out of the watch list.
Keep the join file next to the test output.

What this method refuses to claim

It does not score models.
It does not prove a patch is correct.
It only proves whether the tree moved with the span.

Hash collisions are not the practical risk.
The practical risk is a bad watch list.
Ignore node_modules and build output.
Ignore caches and compiled artifacts.
Watch source and tests only.

Timestamps are not hashes.
Clock skew already mis-orders spans in other writeups.
A content hash still works when clocks drift.
Use nanoseconds only as a duration hint.

Do not store raw file contents in JSONL by default.
Hashes are enough for the join.
Contents belong in git, not in the trace stream.

Limits

This loop fails on generated binaries.
It fails when tools write outside the watch list.
It fails when two tools share one file in one tick.

It also fails for pure network agents.
If the side effect is a ticket or a mail, hash the API body instead.
The same join idea applies. The code above does not.

Path normalization can lie on mixed OS replay.
Store relative POSIX paths in the join file.
Resolve them with pathlib on each host.

Large trees make naive hashing slow.
Cap the watch list.
If you need more files, hash a manifest, not the repo.

Who should not use this

Skip this if you already emit file digests on every OpenTelemetry span.
Skip this if the agent must not read the workspace.
Skip this if the tree holds production secrets.

Skip this for one-shot chat with no tools.
There is no working tree to join.
A message log is enough.

Also skip it when legal logging rules forbid file fingerprints.
A hash of a confidential file is still a fingerprint.
Get a policy review first.

A short checklist before the next agent run

Watch source and tests only.
Hash arguments, results, and selected paths.
Fail the run when a write tool changes nothing.
Re-hash by hand after the span closes.
Compare local and remote join lines.

The agent can still be wrong after a clean join.
That is a patch-content problem.
It is no longer a missing-write problem.

Separate those two bugs.
The dashboard will look less finished.
The tree will finally show up in the trace.

If you need a second machine for the same JSONL join, MonkeyCode’s free server option can host the watcher. Keep the disclosure in mind, and keep the hashes in git.

The Reply Looks Finished. Record the Finish Reason.

Riley Wang — Wed, 16 Sep 2026 03:52:59 +0000

The agent returned only a half-written patch file.
The dashboard still painted a green success check.
Observed latency sat inside the usual service band.
No tool span in the trace had failed.
The model had simply hit max tokens.

Nobody stored a generation finish_reason field.
The exported trace still looked fully complete.
The visible reply was not actually finished.
This is a reconstructed incident, not a customer study.
Treat the numbers below as example fixtures only.

What broke in the run

A coding agent started a small refactor patch.
It opened one search_replace tool call next.
The argument JSON ended in the middle of a key.
The runtime classified that break as a model bug.
The real stop condition was a length cap.

Generation spans still carried status: ok flags.
They omitted finish_reason on every record.
They omitted output_tokens and max_tokens too.
Prompt diffs then blamed the wrong system text.
The budget, not the prompt, cut the bytes.

Why green traces hide length stops

Most agent collectors copy HTTP success only.
The provider returned a normal 200 response.
The token stream closed without a socket error.
Your collector stored the visible output text.
It never stored why generation actually stopped.

Four stop classes matter in daily debugging:

stop: the model chose a natural end
length: the configured token cap fired
tool_calls: it requested tools, not prose
content_filter: the provider blocked the text

Without those labels, later diffs start lying.
A truncated tool call looks like invalid JSON.
A filtered answer looks like a silent model.
A tool_calls stop looks like empty chat output.
Retry counters will not explain any of this.
Clock skew checks will not explain it either.

Artifact: a generation budget record

Do not treat this record as full product telemetry.
Treat it as a required span contract instead.
Every LLM generation writes exactly one record.
Keep the payload small, typed, and portable.

{
  "span_type": "generation",
  "span_id": "gen_8f2c",
  "parent_run_id": "run_19",
  "model_slot": "unspecified-free-slot",
  "finish_reason": "length",
  "max_tokens": 1024,
  "output_tokens": 1024,
  "prompt_tokens": 3188,
  "truncated": true,
  "tool_call_open": true,
  "text_sha256": "c0ffee",
  "ended_at": "2026-09-16T14:02:11Z"
}

Do not invent provider model names in spans.
Store a slot identifier that you control locally.
Map slots outside the trace if names change.
Keep the span format stable across exporters.

truncated must stay a derived boolean field.
Set it when finish_reason equals length.
Also set it when output reaches max_tokens.
Never infer truncation from string length alone.
Unicode, tools, and clippers break that heuristic.

Checker you can run on JSONL

Label this script as an unexecuted example first.
Point it at captured JSONL traces before production.
Fail the run when required fields are missing.

#!/usr/bin/env python3
"""Fail a trace if generation spans omit finish_reason."""
from __future__ import annotations

import hashlib
import json
import sys
from collections import Counter
from typing import Any

REQUIRED = (
    "span_id",
    "finish_reason",
    "max_tokens",
    "output_tokens",
    "truncated",
)

ALLOWED_REASONS = {
    "stop",
    "length",
    "tool_calls",
    "content_filter",
    "unknown",
}


def load_jsonl(path: str) -> list[dict[str, Any]]:
    rows: list[dict[str, Any]] = []
    with open(path, encoding="utf-8") as handle:
        for line_no, line in enumerate(handle, 1):
            line = line.strip()
            if not line:
                continue
            try:
                rows.append(json.loads(line))
            except json.JSONDecodeError as exc:
                raise SystemExit(f"bad jsonl:{line_no}:{exc}") from exc
    return rows


def check_span(span: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    span_id = span.get("span_id", "?")
    for key in REQUIRED:
        if key not in span:
            errors.append(f"{span_id}: missing {key}")
    reason = span.get("finish_reason")
    if reason not in ALLOWED_REASONS:
        errors.append(f"{span_id}: bad reason {reason!r}")
    max_tokens = span.get("max_tokens")
    output_tokens = span.get("output_tokens")
    truncated = span.get("truncated")
    if isinstance(max_tokens, int) and isinstance(output_tokens, int):
        hit_cap = output_tokens >= max_tokens and max_tokens > 0
        if hit_cap and reason not in {"length", "unknown"}:
            errors.append(f"{span_id}: cap without length")
        if truncated is True and reason != "length":
            errors.append(f"{span_id}: truncated flag mismatch")
        if truncated is False and reason == "length":
            errors.append(f"{span_id}: length without truncated")
    return errors


def summarize(spans: list[dict[str, Any]]) -> None:
    reasons = Counter(s.get("finish_reason") for s in spans)
    print("finish_reason counts:")
    for key, value in sorted(reasons.items(), key=lambda kv: (-kv[1], str(kv[0]))):
        print(f"  {key}: {value}")
    truncated = sum(1 for s in spans if s.get("truncated") is True)
    print(f"truncated_generations: {truncated}/{len(spans)}")


def main() -> None:
    if len(sys.argv) != 2:
        raise SystemExit("usage: check_finish_reason.py trace.jsonl")
    rows = load_jsonl(sys.argv[1])
    gens = [r for r in rows if r.get("span_type") == "generation"]
    if not gens:
        raise SystemExit("no generation spans")
    errors: list[str] = []
    for span in gens:
        errors.extend(check_span(span))
    summarize(gens)
    if errors:
        print("contract failures:")
        for item in errors:
            print(f"  {item}")
        raise SystemExit(2)
    print("generation contract ok")


if __name__ == "__main__":
    main()

Run the checker against one captured file.

python3 check_finish_reason.py traces/run_19.jsonl

A failing export often prints this shape.

finish_reason counts:
  None: 4
truncated_generations: 0/4
contract failures:
  gen_8f2c: missing finish_reason
  gen_8f2c: missing max_tokens

Repair the exporter before touching prompts.
Missing fields are collector bugs, not model bugs.
Prompt edits cannot restore an omitted stop class.

Decision table after the contract is green

Use this table before anyone rewrites prompts.

finish_reason	truncated	Next debug move
`stop`	false	Diff prompt text and tool schemas
`length`	true	Raise cap or shrink context first
`tool_calls`	false	Validate tool JSON against schema
`content_filter`	false	Inspect the blocked span text
`unknown`	any	Repair the exporter, then rerun

Most incident reviews reverse this exact order.
They rewrite the system prompt under time pressure.
They add another retry around the same cap.
They never raise max_tokens on that slot.
They never shrink the stale context window.
The next trace still ends in the same cut.

How length stops corrupt tool JSON

Tool calls fail in a specific, repeatable way.
The model starts a JSON object for arguments.
The cap fires before the closing brace arrives.
Your parser raises JSONDecodeError immediately.
The tool span is then marked as a model failure.

That classification sends people to prompt surgery.
The cheaper fix is often a smaller context pack.
Drop unused files from the prompt first.
Lower retrieval count before raising the cap.
Only then change max_tokens on that slot.

Keep a digest even when logs clip the body.
Many collectors truncate long tool payloads.
The digest still proves the bytes changed.
Store the first two kilobytes plus the hash.

def text_digest(text: str) -> str:
    blob = text.encode("utf-8")
    return hashlib.sha256(blob).hexdigest()

If the digest changes and reason is length,
the model did not quietly change its plan.
The configured cap cut the output bytes.
Diff the budget fields before diffing prose.

Attach the fields in one adapter

Keep provider adapters intentionally thin.
Map native stop fields in a single function.
Write the contract inside your collector path.

Proposed mapping notes for an exporter:

Read native finish_reason or stop_reason.
Normalize those values to the five tokens.
Copy usage counts without rounding tricks.
Derive truncated only from those fields.
Hash visible text with SHA-256 as shown.
Persist the record before the next tool call.

Unknown must remain a first-class stored value.
Do not coerce missing reasons into stop.
That coercion recreates the original incident.
Empty usage objects should stay empty, not zero.
Zero tokens look like a finished empty answer.

Where spare compute actually helps

Local JSONL is enough for the checker script.
A shared box helps when trace files grow large.
MonkeyCode is an open-source coding project.
It provides free model access for agent runs.
It also provides a free server option for replay.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use spare compute to replay traces, not scoreboards.
Do not publish unnamed model league tables from this.
Run the checker, then inspect finish_reason counts.
Decide whether the cap, not the prompt, is wrong.
If traces already live in your own store, stay there.
The span contract does not require a new vendor.
The free server is optional batch compute only.
A laptop still runs the same JSONL check cleanly.

Limitations and who should skip this

This contract does not measure answer quality.
A clean stop span can still be factually wrong.
Token counters can lag a partial stream flush.
Some providers omit usage on failed requests.
unknown needs a ticket, not a prompt tweak.

Do not compare models using these raw counts.
Do not turn one-run ratios into ranking charts.
A single trace is a sample size of one.
Clock skew still needs a separate measurement.
Retry counts still need their own span fields.

Skip this approach in these four cases:

Teams that still have no generation logs
Apps that never issue tool calls at all
Brokers that forbid storing token fields
Reviews that only hunt writing style issues

If token counts are blocked by policy rules,
still store finish_reason on every generation.
That one field already splits the debug queue.
Length stops stop looking like prompt failures.

Reusable debug loop

Reuse this loop on the next truncated patch.

Export JSONL that includes generation spans.
Run check_finish_reason.py on that file.
If the contract fails, fix export mapping.
If length dominates, cut context or caps.
If tool_calls dominates, test tool schemas.
Only then diff prompts against prior runs.

The order is the method, not a slogan list.
Skip a step and you will debug ghost failures.
Green HTTP checks are not finish reasons yet.
Record the reason, then change the budget.

Your Trace Shows the Wrong Order. Measure Clock Skew Before You Debug

Riley Wang — Mon, 14 Sep 2026 21:37:48 +0000

Two in the morning, one failing agent run, and a trace that made no sense.

The tool result sat four milliseconds before the tool call that produced it. I read the retry logic for three hours. The retry logic was fine.

The trace was wrong. Two machines wrote it. Their clocks disagreed by about 250 ms.

The bug is in the timeline, not the agent

A single trace often has more than one clock behind it. The agent process stamps spans on its own host. A sandbox or tool server stamps results on another machine.

When those clocks drift, ordering breaks. You then debug a sequence that never happened.

Watch for these symptoms:

A child span ends before its parent starts.
A tool result carries an earlier timestamp than its request.
Server-side durations come out negative.
Skew grows steadily across a long run.
One host's spans all look suspiciously fast.

Any one of these can be a real bug. All of them together usually mean one thing. Your timestamps came from different clocks.

Why two clocks drift apart

The usual causes are boring and common:

Date.now(), time.time(), and friends read a wall clock, not a monotonic one.
Containers share the host clock, but VMs can suspend and resume.
NTP corrections can step a clock forward or backward.
Small rate differences accumulate over hours.

Nothing here is exotic. It is the default in many agent deployments. Measure the offset before you blame the agent.

Do not fix it by re-sorting the trace

Sorting by timestamp hides the problem. The wrong order becomes a consistent wrong order. You lose the signal that told you something was off.

Fix the clocks first. Sort second.

Estimate the offset from paired spans

Every tool call gives you four timestamps:

t1 client sends the request
t2 server receives it
t3 server sends the result
t4 client receives the result

Two numbers fall out of those four.

offset = ((t2 - t1) + (t3 - t4)) / 2
rtt    = (t4 - t1) - (t3 - t2)

This is the classic four-timestamp exchange used by NTP-style sync. It assumes network delay is roughly symmetric. Filter for low-RTT pairs and take the median.

Set max_rtt_ms from your own p50 RTT, not from a blog post. A pair above that threshold tells you about queuing, not about clocks. More pairs help. Aim for dozens, not three.

The estimator

#!/usr/bin/env python3
import json, statistics

def load(path):
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                yield json.loads(line)

def quads(spans):
    by_id = {s['span_id']: s for s in spans}
    for s in spans:
        if s.get('kind') != 'tool_call':
            continue
        r = by_id.get(s.get('result_span_id'))
        if r is None:
            continue
        yield s['ts_ms'], r['ts_ms'], r['end_ms'], s['end_ms']

def split(t1, t2, t3, t4):
    offset = ((t2 - t1) + (t3 - t4)) / 2.0
    rtt = (t4 - t1) - (t3 - t2)
    return offset, rtt

def estimate(qs, max_rtt_ms=50.0, min_pairs=5):
    kept = []
    for t1, t2, t3, t4 in qs:
        off, rtt = split(t1, t2, t3, t4)
        if rtt <= max_rtt_ms:
            kept.append(off)
    if len(kept) < min_pairs:
        raise SystemExit('need %d low-RTT pairs, kept %d' % (min_pairs, len(kept)))
    return statistics.median(kept), len(kept)

if __name__ == '__main__':
    import sys
    spans = list(load(sys.argv[1]))
    off, n = estimate(quads(spans))
    print('offset_ms=%.1f pairs=%d' % (off, n))

A fixture test with a known offset

Never trust an estimator you have not tested against a known answer. Inject an offset, then check recovery.

import random
from spanskew import estimate

def test_recovers_known_offset():
    random.seed(7)
    injected = 250.0
    base = 1_700_000_000_000
    qs = []
    for i in range(40):
        send = base + i * 1000
        up = random.uniform(4, 6)
        down = random.uniform(4, 6)
        work = 5.0
        t2 = send + up + injected
        t3 = t2 + work
        t4 = t3 - injected + down
        qs.append((send, t2, t3, t4))
    off, n = estimate(qs, max_rtt_ms=30.0)
    assert n == 40
    assert abs(off - injected) <= 2.0

The fixture injects a 250 ms offset. The estimator must recover it within 2 ms. If it cannot, your RTT filter or your median is wrong.

Eyeball the raw trace first

One cheap check before any math:

jq -r 'select(.kind=="tool_call") | [.span_id, (.end_ms - .ts_ms)] | @tsv' trace.jsonl | sort -k2 -n | head

Negative or near-zero durations on tool calls are a strong skew hint. So is a result span that sorts before its request span.

A minimal trace line pair looks like this:

{"span_id":"s1","kind":"tool_call","ts_ms":1700000000000,"end_ms":1700000000042,"result_span_id":"s1r"}
{"span_id":"s1r","kind":"tool_result","ts_ms":1700000000250,"end_ms":1700000000255}

Correct the trace, then re-read it

Work in this order:

Group spans by emitting host.
Pick one host as the reference clock.
Compute the median offset for every other host.
Shift those hosts' timestamps by their offset.
Rebuild parent and child links from span ids, not timestamps.
Re-run the original query on corrected stamps.

Step five matters most. Span ids are stable facts. Timestamps are measurements.

Which evidence to trust

Evidence	Trust it when	Fallback
Wall-clock timestamps	Hosts are synced and RTT is low	Corrected offset
Span nesting	Parent ids exist and are unique	Rebuild from ids
Duration fields	Emitted by the same host	Recompute from stamps
Log line order	A single writer, single file	Add sequence numbers

Where to run this harness

The harness is small and CPU-only. It reads JSONL and prints one number. I run it on MonkeyCode's free server option and use free model access to draft the span rules before they land in CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A stable box matters here. Skew work depends on running the same fixture many times. If you want to try the harness without provisioning a machine, the free server option is a reasonable place to start.

Limits, and who should skip this

The symmetry assumption breaks badly under heavy upload or download.
One offset per run fails if NTP steps the clock mid-run.
Fewer than five low-RTT pairs means you have no estimate. Do not guess.
Single-host traces do not need any of this.
If your collector already normalizes clocks, use that instead.
Monotonic clocks fix ordering within one host. They do not fix cross-host comparison.
This tells you nothing about which component is slow. It only fixes the axis.

Skip it entirely if every span comes from one process with one monotonic clock. You have no problem to solve.

A reusable debug loop

Reproduce the run and keep the JSONL trace.
Count the emitters. If there is only one, stop here.
Run the jq check for negative or tiny durations.
Estimate offsets and keep only low-RTT pairs.
Correct stamps, then rebuild links from span ids.
Re-run the failing query on corrected data.

Most of that loop is one script and one jq call. Step one is the step people skip.

A trace is a measurement, not a recording. Measure the ruler before you measure the bug.

Tool Output Looks Complete. Record the Injected Slice.

Riley Wang — Sun, 13 Sep 2026 19:57:19 +0000

The on-call thread opened at 02:11 UTC.
The agent had already marked the run green.
It claimed production.yaml was read and valid.

Slack still received the old staging port.
The file on disk contained the right value.
The trace even stored the entire file body.

That mismatch is the bug this article names.
The tool span was complete and technically truthful.
The next model turn never received that body.

The lie inside a successful span

Most agent logs keep the tool call name.
They also keep arguments and a result string.
Some keep timestamps and a parent span id.

They rarely keep three different length fields.
Those three numbers are not interchangeable.
Mixing them hides a common production failure.

Here are the three lengths that actually matter:

observed_bytes counts bytes the tool returned.
injected_bytes counts bytes copied into context.
injected_tokens counts tokens the API accepted.

A green span can still drop most of a file.
Your replay then debugs the wrong evidence set.
You will rewrite prompts the model never saw.

Clipping happens after the tool returns

This is not a hang and not a retry bug.
The tool finished. The filesystem read succeeded.
The prompt builder then applied a size cap.

Duration fields will look healthy and boring.
Retry counters will stay at one attempt.
Span trees will show a normal child read.

The loss sits on a later boundary.
That boundary is prompt assembly, not I/O.
Traces that stop at tool return will miss it.

Tool success answers one question only.
It says the executor did not throw.
It does not say the model saw the tail.

A reconstructed incident

Treat this section as a reconstructed pattern.
It is not a named customer outage report.

The agent called read_file on production.yaml.
The tool returned 96412 bytes of YAML text.
The trace writer stored the full result blob.

The prompt builder applied an 8192 token cap.
Only the leading keys reached model context.
The port field lived near the file end.

The model answered from that truncated prefix.
It reused a default port from the system prompt.
The trace still showed a perfect file read.

Diffing two full tool bodies did not help.
Both traces contained identical complete files.
Only the injected slice had changed size.

A later run used a tighter builder policy.
Observed hashes stayed stable across both runs.
Injected hashes diverged, and the port flipped.

Artifact: clip-aware JSONL spans

Store clipping as first-class fields on spans.
Do not infer clipping from sampled prompt dumps.
Prompt dumps are often redacted or dropped.

Use one JSON object per line, as follows.

{
  "span_id": "tool-14",
  "parent_id": "turn-3",
  "name": "read_file",
  "status": "ok",
  "observed_bytes": 96412,
  "injected_bytes": 12040,
  "injected_tokens": 8192,
  "clip_reason": "prompt_token_cap",
  "sha256_observed": "ab12",
  "sha256_injected": "cd34",
  "injected_preview": "app:\\n  name: billing\\n",
  "prompt_builder_version": "cap-v3"
}

Keep clip_reason as a small closed enum.
Do not store free-text excuses in that field.

Closed values worth supporting today:

none means the full result was injected.
prompt_token_cap means the builder cut tokens.
tool_result_char_limit means a char cap fired.
provider_context_window means the API rejected tail.
redaction means bytes changed for safety.
binary_omitted means a non-text payload was dropped.

Flag a span when injected bytes fall below observed.
Flag again when hashes differ at equal lengths.
Redaction can change bytes without a size cap.

Checker script

The script below is a local proposal only.
It reads JSONL from stdin and prints a table.
It does not call any network service.

#!/usr/bin/env python3
"""clip_check.py — flag tool spans with silent truncation."""
import json
import sys

PRIORITY = {
    "silent_clip": 0,
    "incomplete": 1,
    "hash_mismatch": 2,
    "clipped": 3,
    "ok": 4,
}


def load_spans(stream):
    for line_no, raw in enumerate(stream, 1):
        line = raw.strip()
        if not line:
            continue
        rec = json.loads(line)
        rec["_line"] = line_no
        yield rec


def clip_ratio(obs, inj):
    if obs <= 0:
        return 0.0
    return 1.0 - (inj / obs)


def classify(rec):
    missing = [
        key
        for key in ("observed_bytes", "injected_bytes", "injected_tokens")
        if rec.get(key) is None
    ]
    if missing:
        return "incomplete", missing

    obs = int(rec.get("observed_bytes") or 0)
    inj = int(rec.get("injected_bytes") or 0)
    reason = rec.get("clip_reason") or "none"

    if inj < obs and reason == "none":
        return "silent_clip", [f"ratio={clip_ratio(obs, inj):.2f}"]
    if inj < obs:
        return "clipped", [reason, f"ratio={clip_ratio(obs, inj):.2f}"]

    obs_hash = rec.get("sha256_observed")
    inj_hash = rec.get("sha256_injected")
    if obs_hash and inj_hash and obs_hash != inj_hash and reason == "none":
        return "hash_mismatch", ["bytes_match_but_hash_differs"]
    return "ok", []


def main():
    rows = []
    for rec in load_spans(sys.stdin):
        if not rec.get("name"):
            continue
        status, detail = classify(rec)
        rows.append((status, rec, detail))

    rows.sort(key=lambda item: PRIORITY.get(item[0], 9))
    print(f"{'status':<16} {'span':<12} {'tool':<16} {'obs':>8} {'inj':>8} detail")
    for status, rec, detail in rows:
        print(
            f"{status:<16} {str(rec.get('span_id', '-')):<12} "
            f"{str(rec.get('name', '-')):<16} "
            f"{int(rec.get('observed_bytes') or 0):>8} "
            f"{int(rec.get('injected_bytes') or 0):>8} "
            f"{','.join(detail)}"
        )

    silent = sum(1 for status, _, _ in rows if status == "silent_clip")
    incomplete = sum(1 for status, _, _ in rows if status == "incomplete")
    print(f"\n# silent_clip={silent} incomplete={incomplete} total={len(rows)}")
    if silent or incomplete:
        sys.exit(1)


if __name__ == "__main__":
    main()

Run it against one exported run:

python3 clip_check.py < traces.jsonl
echo $?

Exit code 1 means the trace is not trustworthy.
Exit code 0 means every tool span declared clipping.
That zero does not prove the answer was correct.

Emit fields at the tool boundary

Wrap the executor so spans cannot omit lengths.
Hash both strings before the prompt is built.
Write the JSONL line before the next model call.

import hashlib
import json
from pathlib import Path


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def emit_tool_span(path, span_id, parent_id, name, observed, injected, reason, builder_version, token_count):
    rec = {
        "span_id": span_id,
        "parent_id": parent_id,
        "name": name,
        "status": "ok",
        "observed_bytes": len(observed.encode("utf-8")),
        "injected_bytes": len(injected.encode("utf-8")),
        "injected_tokens": token_count,
        "clip_reason": reason,
        "sha256_observed": sha256_text(observed),
        "sha256_injected": sha256_text(injected),
        "injected_preview": injected[:128],
        "prompt_builder_version": builder_version,
    }
    with Path(path).open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(rec, ensure_ascii=False) + "\n")
    return rec

Call emit_tool_span once per finished tool.
Pass the exact string the prompt builder used.
Do not hash the file on disk after the fact.

Disk can change between tool return and replay.
The injected string is the only fair evidence.
That is what the model could have used.

If your builder truncates after encoding tokens, record that.
Byte cuts and token cuts are different operations.
The enum must say which policy actually fired.

Four-step debug loop

Use the same loop on every flaky agent run.
Do not start with prompt rewriting first.

Export one JSONL file per agent run.
Require clip fields on every tool span.
Fail the run if clip_check.py exits 1.
Replay only after injected hashes match.

Step four needs the injected slice, not the file.
Keep injected_preview as a short stable prefix.
A 128-byte prefix is enough to compare two runs.

Add these extra fields when you can:

injected_preview stores 128 escaped leading bytes.
model_context_limit_tokens stores the active window.
prompt_builder_version stores the cap policy id.

prompt_builder_version catches silent policy changes.
A new cap can ship without any model change.
Your model regression may be a clip change.

Compare builder versions before you compare models.
A cap change rewrites every downstream token.
Model swap notes become noise after that rewrite.

Tests for the classifier

These tests are local and do not need GPUs.
They lock the silent-clip definition in place.
Run them before you trust a new exporter.

from clip_check import classify


def test_silent_clip():
    status, detail = classify({
        "observed_bytes": 1000,
        "injected_bytes": 120,
        "injected_tokens": 80,
        "clip_reason": "none",
    })
    assert status == "silent_clip"
    assert detail[0].startswith("ratio=")


def test_declared_clip_is_visible():
    status, _ = classify({
        "observed_bytes": 1000,
        "injected_bytes": 120,
        "injected_tokens": 80,
        "clip_reason": "prompt_token_cap",
    })
    assert status == "clipped"


def test_missing_lengths_fail():
    status, missing = classify({"name": "read_file"})
    assert status == "incomplete"
    assert "observed_bytes" in missing

A silent clip is the highest priority failure.
Missing fields are the next failure class.
Declared clips are visible and therefore usable.

Do not fail CI on every declared clip.
Large read_file results will clip by design.
Fail CI when the reason field is missing or none.

Decision table

Symptom	Trace looks like	First check	Do not do
Wrong field from a large file	`status=ok`, full body stored	`injected_bytes` versus `observed_bytes`	Rewrite the system prompt
Binary tool result looks empty	short ASCII replacement text	`clip_reason=binary_omitted`	Retry the same tool blindly
Intermittent wrong answers	identical observed tool bodies	builder version and cap fields	Swap models immediately
Replay missing expected PII	hashes differ, lengths stay close	`clip_reason=redaction`	Paste raw traces into chat
Checker exits incomplete	length fields are absent	fix the exporter first	Trust the pretty trace UI

Wrong-port bugs often match row one.
The file was complete in object storage.
The injected slice never included the key.

Where a quiet box fits

You can run this loop on a laptop.
A quiet remote box helps with overnight replays.
Long traces should not compete with your IDE.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source project for coding agents.
It offers free model access and a free server option.

Those two facts matter for this workflow only.
You need a model to regenerate injected slices.
You need a server if local machines are busy.

Do not treat that option as a benchmark lab.
This article does not publish model names.
It also does not publish quotas or hardware specs.

Capture traces. Run clip_check.py. Then replay.
If the checker is red, skip model comparison.
You would be scoring two different prompts.

If you already store JSONL traces, start there.
The server is optional for the checker itself.

Limitations

This method does not score answer quality.
It only detects undeclared length or hash changes.
A fully injected file can still be the wrong file.

Token counts depend on the provider tokenizer.
Byte ratios are exact. Token ratios are estimates.
Do not alert on token estimates alone.

Streaming tools can update observed bytes twice.
Record final values after the tool span closes.
Partial spans should keep status as incomplete.

Multi-agent graphs need per-child clip fields.
A parent ok can hide a clipped child read.
The checker should walk every tool span record.

Pretty UIs often show the observed blob first.
Engineers then believe the model saw that blob.
Show the injected slice before the observed body.

This loop also ignores semantic irrelevance.
A full injection can still be the wrong excerpt.
You still need evals for that class of error.

Who should not use this

Skip this if your agent has no tools.
A chat wrapper has no result slice to clip.
Prompt-only apps need a different trace shape.

Skip this if traces contain raw secrets.
Hash and preview fields can still leak data.
Redact before JSONL ever leaves the host.

Skip this for hard real-time control loops.
The extra fields add exporter complexity and I/O.
Those systems need different runtime guarantees.

What to watch after you ship this

Once clipping is explicit, add one rate.
Count silent clips per one hundred tool spans.
Watch that rate when prompt builders change.

A second watch is useful after the first.
Track median injected bytes for read_file.
A sudden drop often ships with a new cap.

Keep the original failure in mind here.
The YAML file was on disk and correct.
The model never saw the port key at all.

Record the injected slice on every tool span.
Then green traces stop lying about completed reads.

Agent Traces Hide the Fork. Record Ranked Options.

Riley Wang — Sat, 12 Sep 2026 17:55:58 +0000

A coding agent returned a green run at 02:11.
The ticket asked for a small test helper.
This scene is a constructed on-call example.

The agent called search_repo and then edit_file.
Both tool spans closed with status ok.
The patch landed in the wrong Python package.

The JSONL export listed every call in order.
It never listed the tools the planner rejected.
Ops could not replay the fork that mattered.

Execution logs are not routing logs

Most agent traces are still plain call journals.
They store name, arguments, latency, and final status.
They omit the ranked options behind the pick.

A green tool span answers one narrow question.
It only asks if the invocation threw an error.
It does not answer why this tool won.

Loop-style agents fail at the fork often.
The wrong search query still returns hits.
The wrong file still accepts a clean edit.

Those runs look healthy in span dashboards.
The user still receives a useless change.
You need a record of discarded alternatives.

What a decision span must capture

Record the choice before the tool starts.
Do not reconstruct intent from later arguments.
Arguments describe the winner, not the field.

Each decision span needs a stable shape.
Keep it small enough to grep in JSONL.
Keep it strict enough to validate in CI.

Use these minimum fields on every decision span:

Store span_id and parent_id for tree joins.
Store ts in UTC with second precision.
Store goal as a short remaining task.
Store options as scored tool candidates.
Store chosen matching one option name.
Store policy naming the selection rule.
Store margin as winner minus runner-up.

Skip free-text essays in this span.
Put long prompt blobs in a separate event.
Decision records should stay cheap to diff.

A concrete JSONL fragment

Example below is illustrative, not a live export.

{"type":"decision","span_id":"dec_7f3a","parent_id":"run_91","ts":"2026-09-13T02:11:04Z","goal":"add a date-parse test helper","options":[{"tool":"search_repo","score":0.81,"reason":"find existing helpers"},{"tool":"edit_file","score":0.44,"reason":"patch a remembered path"},{"tool":"run_tests","score":0.12,"reason":"no diff exists yet"}],"chosen":"search_repo","policy":"argmax_score","margin":0.37}
{"type":"tool_call","span_id":"call_12","parent_id":"dec_7f3a","ts":"2026-09-13T02:11:05Z","tool":"search_repo","args":{"query":"date parse helper"}}
{"type":"tool_result","span_id":"res_12","parent_id":"call_12","ts":"2026-09-13T02:11:06Z","ok":true,"bytes":842}

Notice the parent link on the tool call.
The call hangs under the decision, not the run.
That join is the entire debugging upgrade.

Validator you can run on a file

The next script is a local example.
It is not a measured production harness.
Save it as check_decisions.py beside a JSONL trace.

#!/usr/bin/env python3
"""Flag tool calls that lack a scored decision."""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from typing import Any


@dataclass
class Finding:
    line: int
    kind: str
    detail: str


def load_events(path: str) -> list[tuple[int, dict[str, Any]]]:
    rows: list[tuple[int, dict[str, Any]]] = []
    with open(path, encoding="utf-8") as handle:
        for line_no, raw in enumerate(handle, start=1):
            text = raw.strip()
            if not text:
                continue
            rows.append((line_no, json.loads(text)))
    return rows


def margin_of(options: list[dict[str, Any]], chosen: str) -> float | None:
    scores = {str(item["tool"]): float(item["score"]) for item in options}
    if chosen not in scores or len(scores) < 2:
        return None
    winner = scores[chosen]
    rest = [value for name, value in scores.items() if name != chosen]
    return winner - max(rest)


def audit(path: str) -> list[Finding]:
    findings: list[Finding] = []
    pending: dict[str, dict[str, Any]] = {}
    for line, event in load_events(path):
        kind = event.get("type")
        if kind == "decision":
            options = event.get("options") or []
            chosen = str(event.get("chosen") or "")
            tools = [str(item.get("tool")) for item in options]
            if chosen not in tools:
                findings.append(Finding(line, "chosen_missing", chosen))
            computed = margin_of(options, chosen)
            reported = event.get("margin")
            if computed is not None and reported is not None:
                if abs(float(reported) - computed) > 1e-6:
                    findings.append(Finding(line, "bad_margin", str(reported)))
            pending[str(event.get("span_id"))] = event
            continue
        if kind == "tool_call":
            parent = str(event.get("parent_id") or "")
            tool = str(event.get("tool") or "")
            decision = pending.pop(parent, None)
            if decision is None:
                findings.append(Finding(line, "missing_decision", tool))
                continue
            if str(decision.get("chosen")) != tool:
                findings.append(
                    Finding(line, "choice_mismatch", f"{decision.get('chosen')}->{tool}")
                )
    for leftover in pending.values():
        findings.append(
            Finding(0, "unconsumed_decision", str(leftover.get("span_id")))
        )
    return findings


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: check_decisions.py trace.jsonl", file=sys.stderr)
        return 2
    findings = audit(sys.argv[1])
    for item in findings:
        print(f"{item.kind}\tL{item.line}\t{item.detail}")
    print(f"finding_count\t{len(findings)}")
    return 1 if findings else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it against a tiny fixture first.

python3 check_decisions.py sample.jsonl

A clean file prints a zero finding count.
A missing parent prints the missing_decision kind.
A rewritten winner prints the choice_mismatch kind.

A debug loop that diffs forks

Do not start with model quality debates.
Start with two traces of the same ticket.
Keep the user prompt identical on both runs.

Work the loop in this fixed order:

Confirm both JSONL files parse line by line.
Extract the sequence of chosen tools only.
Extract the option sets sitting under each choice.
Flag any tool call without a parent decision.
Flag margins below a local threshold, such as 0.05.
Diff the two chosen sequences as plain text.
Only then inspect arguments on the first mismatch.

Example commands for steps two and six:

jq -r 'select(.type=="decision") | .chosen' a.jsonl > a.chosen.txt
jq -r 'select(.type=="decision") | .chosen' b.jsonl > b.chosen.txt
diff -u a.chosen.txt b.chosen.txt

If the chosen lists match, the bug is downstream.
If they diverge, stop reading tool payloads.
Read the option scores at the first split.

Low margin is a routing smell, not a proof.
A 0.02 gap means the planner was almost tied.
Tied planners need a policy, not another retry.

How this differs from call diffs

Call diffs tell you arguments changed later.
Decision diffs tell you the menu changed first.
Those are different bugs with different fixes.

A search query tweak is an argument bug.
A missing search_repo option is a prompt bug.
A tiny margin is a scoring bug.

Treat those three cases as separate tickets.
Mixing them produces noisy agent evaluation reports.
Routing quality is not the same as tool quality.

Where a scratch model host helps

You can run the validator on a laptop.
Replaying the agent still needs a model endpoint.
It also needs a quiet machine for two traces.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode currently offers free model access and a free server option.
Use that pair only as a scratch replay host.

They are not a substitute for a decision schema.
Add the spans first, then replay on any host you already use.
Keep the product choice secondary to the span contract.

Capture two JSONL files for the same ticket.
Compare chosen lists before you compare patches.
Stop at the first fork that changed rank.

Limitations

Model-supplied scores are not calibrated success probabilities.
Do not treat 0.81 as an 81 percent success chance.
Use scores only to rank options inside one span.

Option reason fields can leak private context.
Redact paths, tokens, and customer names before storage.
Drop reasons in production if your store is wide.

This pattern adds one event per tool call.
Busy loops will grow trace volume quickly.
Sample long exploratory runs, never the decision keys.

Clock skew still disorders events across hosts.
The validator assumes a single writer per file.
Do not merge JSONL from two clocks without ids.

Who should not use this

Skip this if the agent has one hard-coded tool.
There is no fork, so ranking adds noise.

Skip this if you cannot store traces at all.
A schema without a file is documentation theater.

Skip this for binary policy routers with no scores.
Log the rule name, not a fake numeric margin.

Skip this when option text includes secrets.
Fix redaction before you add richer spans.

What to keep in the next incident

When a green agent still ships a bad edit, ask four questions.

Did every tool call cite a decision parent?
Did chosen match the tool that actually ran?
Was the margin too small to trust the pick?
Did a second run diverge at the same fork?

If you cannot answer those from the file, the trace is incomplete.
Call journals will keep looking green under dashboards.
The rejected options will keep disappearing from review.

Ship the schema before you shop for another dashboard.
A ranked option list is a small, testable artifact.
It survives even if you change models next week.

Green Agent Loops Never End. Require a Terminal Span

Riley Wang — Fri, 11 Sep 2026 15:39:25 +0000

Here is a reconstructed staging incident from logs.
A staging bot was asked to open one pull request.
Every recorded tool span still reported status ok.

The promised pull request never appeared on GitHub.
The run lasted forty-one minutes after the last write.
Logs kept growing while the heartbeats stayed pretty.

No span ever said the loop was finished.
Operators marked the job successful anyway.
Success meant "no exception," not "goal met."

Green spans are not a stop signal

Most agent traces record tools instead of goals.
A read_file span can succeed one hundred times.
That success does not mean the job completed.

Dashboards hide the gap behind a zero error rate.
People watch errors and miss runs that never stop.
The process can stay busy until someone kills it.

You need a contract the trace can fail.
The failure contract remains simple and locally testable.
Every run must emit one terminal span.

What a terminal span must prove

A terminal span is not another tool result.
It names why the loop stopped acting.
It also carries the iteration that stopped.

Accept only these stop reasons at first:

done means the stated goal was actually met
rejected means a guard blocked the next act
budget means iteration or time ran out
blocked means a tool returned a hard failure
handoff means another run now owns the goal

If the stream ends without one reason, the run is open.
Open runs are incidents, not quiet successes.
Treat them like missing health checks on a service.

Decision table for a single run

Observation	Class	Action
last span is a tool with `ok`	`open_loop`	cap the run and page
terminal `done` and iter within budget	`closed_ok`	keep a sample only
terminal reason equals `budget`	`closed_cap`	inspect the last plan
two terminal spans in one run	`double_stop`	fix the emitter
iteration sequence contains gaps	`bad_clock`	sort by `seq`, not time
same `args_hash` for three iterations	`no_progress`	halt on the next match

Do not fold these classes into HTTP status codes.
A tool HTTP 200 is not done.
The done flag is a claim about the goal.

JSONL schema you can emit tonight

Keep one JSON object on each line.
Do not pretty-print objects inside the file.
Stable keys beat nested vendor blobs every time.

{"seq":1,"ts":"2026-09-11T02:00:01Z","run_id":"run_7f3","span_id":"sp_01","type":"loop_start","iter":0,"goal":"open_pr"}
{"seq":2,"ts":"2026-09-11T02:00:03Z","run_id":"run_7f3","span_id":"sp_02","parent_id":"sp_01","type":"tool","iter":1,"name":"read_file","args_hash":"a91c","result_hash":"b33e","status":"ok"}
{"seq":3,"ts":"2026-09-11T02:00:08Z","run_id":"run_7f3","span_id":"sp_03","parent_id":"sp_01","type":"tool","iter":2,"name":"edit_file","args_hash":"c10d","result_hash":"c10d","status":"ok"}
{"seq":4,"ts":"2026-09-11T02:41:12Z","run_id":"run_7f3","span_id":"sp_88","parent_id":"sp_01","type":"terminal","iter":38,"reason":"budget","status":"ok","summary":"no_pr_url"}

The seq field is the order the emitter observed.
Do not trust wall clocks across two hosts.
The args_hash field is a short digest of arguments.

Never write secrets into this trace file.
Hash payloads and drop tokens before flush.
A debug loop that leaks keys is worse than silence.

Reproducible checker

The script below is a local proposal.
The script reads JSONL records from standard input.
It prints one verdict line for each run_id.

Treat the listing as an unexecuted local example.
Wire your own emitter before you trust results.

#!/usr/bin/env python3
"""Fail closed when an agent run never terminals."""
from __future__ import annotations

import json
import sys
from collections import defaultdict

BUDGET = 24
NO_PROGRESS_WINDOW = 3
TERMINAL_REASONS = {"done", "rejected", "budget", "blocked", "handoff"}
FAIL = {
    "open_loop",
    "no_progress",
    "double_stop",
    "bad_terminal",
    "bad_clock",
}


def load_spans(stream):
    runs = defaultdict(list)
    for line_no, raw in enumerate(stream, 1):
        text = raw.strip()
        if not text:
            continue
        span = json.loads(text)
        span["_line"] = line_no
        runs[span["run_id"]].append(span)
    return runs


def tool_arg_hashes(spans):
    values = []
    for span in spans:
        if span.get("type") != "tool":
            continue
        values.append(span.get("args_hash") or "")
    return values


def classify(spans):
    ordered = sorted(spans, key=lambda item: item.get("seq", item["_line"]))
    iters = [item.get("iter") for item in ordered if item.get("iter") is not None]
    terminals = [item for item in ordered if item.get("type") == "terminal"]
    max_iter = max(iters) if iters else 0

    if len(terminals) > 1:
        return "double_stop", max_iter

    for left, right in zip(iters, iters[1:]):
        if left + 1 < right:
            return "bad_clock", max_iter

    chain = tool_arg_hashes(ordered)
    if len(chain) >= NO_PROGRESS_WINDOW:
        tail = chain[-NO_PROGRESS_WINDOW:]
        if tail[0] and len(set(tail)) == 1:
            return "no_progress", max_iter

    if not terminals:
        return "open_loop", max_iter

    reason = terminals[0].get("reason")
    if reason not in TERMINAL_REASONS:
        return "bad_terminal", max_iter
    if reason == "done" and max_iter <= BUDGET:
        return "closed_ok", max_iter
    if reason == "budget":
        return "closed_cap", max_iter
    return f"closed_{reason}", max_iter


def main() -> int:
    failed = 0
    runs = load_spans(sys.stdin)
    for run_id, spans in sorted(runs.items()):
        verdict, max_iter = classify(spans)
        print(f"{run_id}\t{verdict}\titer={max_iter}\tspans={len(spans)}")
        if verdict in FAIL:
            failed = 1
    return failed


if __name__ == "__main__":
    raise SystemExit(main())

Save and run

Save the file as terminal_contract.py beside fixtures.
Pipe a known-bad trace through it first.
Confirm a non-zero exit before wiring CI.

python3 terminal_contract.py < traces.jsonl
echo $?

A non-zero exit means some run violated the contract.
Put that exit code on recorded fixtures in CI.
Do not wait for a human to scroll JSONL by hand.

Known-bad fixture

This tiny fixture mimics the staging incident.
There is no terminal span in the stream.
Iteration three already repeats the same edit hash.

{"seq":1,"run_id":"run_bad","type":"loop_start","iter":0,"goal":"open_pr"}
{"seq":2,"run_id":"run_bad","type":"tool","iter":1,"name":"edit_file","args_hash":"aaaa","status":"ok"}
{"seq":3,"run_id":"run_bad","type":"tool","iter":2,"name":"edit_file","args_hash":"aaaa","status":"ok"}
{"seq":4,"run_id":"run_bad","type":"tool","iter":3,"name":"edit_file","args_hash":"aaaa","status":"ok"}

Expected classification is no_progress for run_bad.
The checker should exit with a non-zero status.
If it prints closed_ok, your emitter is lying.

Where a quiet server actually helps

The checker itself is CPU-light and sequential.
It needs a box that stays up between agent runs.
A sleeping laptop will drop the JSONL tail.

MonkeyCode's free server option fits that collector role.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Park the tailer there and keep fixtures beside it.

Free model access is optional on the same box.
Use a model only after the contract already failed.
Ask it to cluster summary strings, not to invent spans.

Do not send raw tool payloads to any model.
Send hashes, tool names, and stop reasons only.
The verdict must still come from the script.

A debug loop you can reuse

Emit loop_start with an explicit goal field.
Stamp every tool span with iter plus args_hash.
Emit exactly one terminal span when work stops.
Run terminal_contract.py against the JSONL file.
Freeze the run on open_loop or no_progress.
Label leftover summaries with a model last.

This loop does not replace a tracing backend.
It is a closed-world test on your own JSONL.
Start here, then export spans if files grow.

What this does not catch

The contract cannot see side effects outside the trace.
A done span can still lack a real pull request.
Verify the artifact with a separate probe each time.

Short hashes can hide real argument drift under collision.
Use a longer digest if tests ever collide.
Do not treat four hex characters as a security boundary.

Wall-clock sorting will mis-order concurrent emitters.
That is why seq exists in the schema.
If seq is missing, the checker becomes a guess.

Parallel tools may share one iteration on purpose.
The gap check allows repeated iteration numbers on purpose.
It only flags skipped numbers in the sequence.

This is also not retry accounting on one tool.
Retries share one tool goal and one family of attempts.
Loop stalls keep planning and still go nowhere new.

Who should skip this approach

Skip this if your runs have no stable run_id.
Skip this if legal holds need immutable audit stores.
Skip this if traces still contain live credentials.

Multi-tenant platforms need redaction before any JSONL drop.
This script does not redact fields for you.
Add that layer first, or do not collect traces.

Teams without a hard iteration budget will ignore verdicts.
Pick a number and write it in the file header.
A budget you never enforce is decoration on logs.

Practical next step

Add type=terminal to one agent tonight.
Replay last week's traces if those files still exist.
Count how many green jobs never actually stopped.

Retries Look Green. Count Attempts on Every Tool Span

Riley Wang — Wed, 09 Sep 2026 15:50:52 +0000

The checkout agent returned HTTP 200 to the storefront. Finance later found two captures on one order.

Support exported one successful tool result from logs. The payment processor listed two authorized captures instead.

The span looked healthy because the last attempt won. Earlier attempts still spent tokens and side effects.

Last Success Is Not One Attempt

Many agent runtimes keep only the final tool payload. Retry wrappers often swallow the failed calls underneath.

Logs then show one green span and one result. Reality contains extra network calls and extra token bursts.

This is not an orphan result or missing pairing bug. The final result exists, yet attempts were never counted.

Green status answers a different question than attempt count. Status asks whether the last try worked. Count asks how much work hid beneath that win.

Three Retry Layers, One Log Line

Duplicate charges rarely come from one wrapper. Three layers retry without sharing a counter.

Transport libraries retry on reset and timeout.
Tool executors retry when JSON arguments fail validation.
The agent retries the whole step after a watchdog timeout.

Each layer can emit success on the last try. None of them must emit the earlier tries.

If exporters flatten those layers into one event, storms vanish. Billing systems still see every side effect.

What To Record Per Attempt

Store one event per try, not per finished span. Keep a stable span_id across those numbered tries.

Use these fields on every retry-aware tool event:

run_id: one agent invocation
span_id: one logical tool call
attempt: integer starting at 1
tool_name: stable tool identifier
args_hash: SHA-256 of canonical JSON arguments
idem_hash: SHA-256 of the provider idempotency key
status: ok, error, or timeout
t_start and t_end: unix milliseconds
token_in and token_out: integers or null
side_effect: none, unknown, or external

Skip raw argument bodies in the shared index. Hash them and keep payloads in a local file.

Why hash arguments

Payment tools often carry account numbers and names. Full payloads do not belong on a shared disk.

A hash still joins retries of the same call. You can group storms without leaking customer PII.

Hash the idempotency key the same way. The raw key still goes to the provider. The trace only needs a joinable digest.

Example Timeline For One Span

The next block is a labeled example, not production evidence. Three attempts share span_19 and one argument hash.

{"event":"tool_attempt","run_id":"run_7f3","span_id":"span_19","attempt":1,"tool_name":"charge_card","args_hash":"a91c9e0c","idem_hash":"e2b1aa44","status":"timeout","t_start":1757385600120,"t_end":1757385604120,"token_in":812,"token_out":0,"side_effect":"unknown"}
{"event":"tool_attempt","run_id":"run_7f3","span_id":"span_19","attempt":2,"tool_name":"charge_card","args_hash":"a91c9e0c","idem_hash":"e2b1aa44","status":"error","t_start":1757385604200,"t_end":1757385606100,"token_in":790,"token_out":48,"side_effect":"unknown"}
{"event":"tool_attempt","run_id":"run_7f3","span_id":"span_19","attempt":3,"tool_name":"charge_card","args_hash":"a91c9e0c","idem_hash":"e2b1aa44","status":"ok","t_start":1757385606200,"t_end":1757385607800,"token_in":788,"token_out":36,"side_effect":"external"}

Attempt three is the only ok row. A last-write exporter would keep that row alone.

The first two rows still consumed tokens and network time. If the provider ignored idem_hash, two captures can exist.

Score Completeness Before Debugging

Do not start a root-cause hunt on thin traces. Score the file first, then inspect storms.

Required keys for this workflow:

run_id
span_id
attempt
tool_name
args_hash
status
t_start
t_end

A run with missing attempt scores as incomplete. Treat incomplete runs as untrusted for cost analysis.

Completeness is a gate, not a verdict. A full record can still describe a wrong tool.

Artifact: Retry Storm Scanner

The script below is a local example. It reads JSONL from stdin. It does not open a network socket.

It groups events by span_id. It flags spans with more than one attempt. It also flags status flips inside one span.

#!/usr/bin/env python3
"""Retry storm scanner for agent tool traces.

Example only. Not production telemetry.
"""
from __future__ import annotations

import hashlib
import json
import sys
from collections import defaultdict
from typing import Any, Iterator

REQUIRED = (
    "run_id",
    "span_id",
    "attempt",
    "tool_name",
    "args_hash",
    "status",
    "t_start",
    "t_end",
)


def iter_events(stream) -> Iterator[dict[str, Any]]:
    for line_no, raw in enumerate(stream, 1):
        raw = raw.strip()
        if not raw:
            continue
        try:
            event = json.loads(raw)
        except json.JSONDecodeError as exc:
            print(f"skip line {line_no}: {exc}", file=sys.stderr)
            continue
        yield event


def completeness(event: dict[str, Any]) -> float:
    present = sum(
        1 for key in REQUIRED if key in event and event[key] not in (None, "")
    )
    return present / len(REQUIRED)


def canonical_args_hash(args: Any) -> str:
    blob = json.dumps(args, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()


def scan(events: Iterator[dict[str, Any]], storm_threshold: int = 2) -> dict[str, Any]:
    by_span: dict[str, list[dict[str, Any]]] = defaultdict(list)
    incomplete = 0
    total = 0
    for event in events:
        if event.get("event") != "tool_attempt":
            continue
        total += 1
        if completeness(event) < 1.0:
            incomplete += 1
            continue
        by_span[str(event["span_id"])].append(event)

    storms = []
    flips = []
    for span_id, attempts in by_span.items():
        attempts.sort(key=lambda row: int(row["attempt"]))
        n = len(attempts)
        statuses = [row["status"] for row in attempts]
        hashes = {row["args_hash"] for row in attempts}
        first = attempts[0]
        last = attempts[-1]
        if n >= storm_threshold:
            storms.append(
                {
                    "span_id": span_id,
                    "run_id": first["run_id"],
                    "tool_name": first["tool_name"],
                    "attempts": n,
                    "statuses": statuses,
                    "unique_args": len(hashes),
                    "ms": int(last["t_end"]) - int(first["t_start"]),
                    "idem_hash": first.get("idem_hash"),
                }
            )
        if len(set(statuses)) > 1:
            flips.append(span_id)

    storms.sort(key=lambda row: row["attempts"], reverse=True)
    return {
        "tool_events": total,
        "incomplete_events": incomplete,
        "spans": len(by_span),
        "storm_spans": len(storms),
        "status_flip_spans": len(flips),
        "top_storms": storms[:20],
    }


def main() -> None:
    report = scan(iter_events(sys.stdin))
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main()

Dry-run commands

Save the three example rows as attempts.jsonl. Then run the scanner from the project directory.

python3 -m py_compile retry_storm.py
python3 retry_storm.py < attempts.jsonl

Expected report shape for that fixture, not a live benchmark:

{
  "tool_events": 3,
  "incomplete_events": 0,
  "spans": 1,
  "storm_spans": 1,
  "status_flip_spans": 1
}

If incomplete_events is not zero, stop. Fix the exporter before reading top_storms.

Emit Attempts At The Tool Boundary

The wrapper below is proposed example code. It is not a measured production client.

import time
from typing import Any, Callable


def call_with_attempts(
    emit,
    run_id: str,
    span_id: str,
    tool_name: str,
    args: dict[str, Any],
    fn: Callable[[dict[str, Any]], Any],
    *,
    attempts: int = 3,
    idem_hash: str,
    args_hash: str,
    side_effect: str = "unknown",
) -> Any:
    last_exc: Exception | None = None
    for attempt in range(1, attempts + 1):
        t_start = int(time.time() * 1000)
        status = "ok"
        result = None
        try:
            result = fn(args)
        except TimeoutError:
            status = "timeout"
            last_exc = TimeoutError(tool_name)
        except Exception as exc:  # example only; narrow this in real code
            status = "error"
            last_exc = exc
        t_end = int(time.time() * 1000)
        emit(
            {
                "event": "tool_attempt",
                "run_id": run_id,
                "span_id": span_id,
                "attempt": attempt,
                "tool_name": tool_name,
                "args_hash": args_hash,
                "idem_hash": idem_hash,
                "status": status,
                "t_start": t_start,
                "t_end": t_end,
                "token_in": None,
                "token_out": None,
                "side_effect": side_effect if status == "ok" else "unknown",
            }
        )
        if status == "ok":
            return result
    raise last_exc if last_exc else RuntimeError("tool failed")

Pass one span_id into every retry of that logical call. Do not mint a new span for attempt two.

Leave token_in null when the runtime hides usage. Null is better than a fabricated zero.

Classify A Storm Before You Patch

When the scanner prints a storm, classify it first. The table is a decision aid, not a proof.

Observation	Likely cause	First check
`unique_args = 1`, mixed `error`/`ok`	transient dependency	timeout budget and idempotency key
`unique_args > 1`	agent changed arguments	schema, validator, prompt
many attempts, all `ok`	retry after success	success predicate
high `incomplete_events`	exporter bug	required field list
same `idem_hash`, two provider captures	provider ignored the key	key header and provider docs

Do not start with a model swap. Most storms are counters, predicates, or keys.

Sampling Without Dropping Storms

Full traces grow faster than useful signal. Sample clean spans. Keep every storm span.

A simple local policy looks like this:

keep 100% of spans with attempts >= 2
keep 100% of spans with status != ok
keep 5% of single-attempt ok spans
drop raw payloads after the hash exists

The 5% figure is an example knob. Tune it against disk, not against a marketing quota.

Where Free Model Access Fits

Some storms need a second pass over redacted error strings. A model can cluster those strings after hashes exist.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access. It also offers a free server option for small trace indexes.

Use the model on redacted messages only. Keep hashes and counts on the server. Keep raw tool bodies on the laptop.

The scanner still works without that product. Stdin and JSONL are enough for the count.

If those free options are already in reach, point them at storm summaries. Do not upload card numbers or raw keys.

Limitations

Truncated hashes can collide, though full SHA-256 is usually enough. Treat short prefixes as debug labels only.

side_effect=unknown is common and weakens billing conclusions. Clock skew across hosts can inflate ms.

The scanner ignores streaming token events by design. Completeness scoring does not prove a tool was correct.

Example thresholds are not capacity plans. They are starting gates for a local debug loop.

Who Should Skip This

Skip this workflow if tools never retry. A counter on single-attempt spans adds noise.

Skip it under an extreme threat model that forbids argument hashes. Some regulated payloads cannot be digested off-box.

Skip it for hard real-time loops that cannot buffer JSONL. This method assumes a file you can score later.

Do not treat the script as a managed APM replacement. It does not trace processes, GC, or kernel waits.

Tonight's Checklist

Emit attempt on every tool try.
Reuse one span_id across those tries.
Hash arguments before any shared disk.
Score completeness on ingest.
Alert on storm_spans, not only on errors.

Green spans can still charge twice. Count attempts before you trust the trace.