Call the Agent Fixed Only After a Bounded Trace Diff

A second agent run that prints a healthy answer is not a fix. The only evidence that a prompt, a tool, or a retry policy changed the system in a controlled way is a bounded diff of spans, tool calls, and logs that still join to a span id.

Stdout is a press release. Traces are the ledger. Most agent failures do not raise. They insert an extra search, rewrite one argument, move a parent, or emit a warning that no longer belongs to any span. If you only compare final text, you accept a different machine and call it the same job.

This article treats the pairwise trace diff as the regression oracle. The artifact is a small recorder, a canonical digest, and a classifier that fails a “green” rerun when the change set is unbounded. It is a proposal you can run locally. It is not a claim about production telemetry you have not measured.

Why the second run lies

An agent loop is closer to a git working tree than to a function. Each tool call is a commit. Each nested span is a branch. Logs are comments that only count if they point at a commit hash. Re-running until the last line looks right is like squashing history because the README matches.

Timestamps make the lie worse. Two runs on a free or loaded endpoint will not share wall clocks, so sorting by time invents order. Causal structure is the parent id, the tool sequence, and the argument digest. Those three survive jitter. Wall time does not.

A useful debug loop therefore stores a failing capture, applies one change, stores a second capture, and asks a narrow question: which trace facts moved, and were those moves on an allowlist? If the answer is “the text got better,” you do not yet have a fix. You have a different story.

A canonical record, not a novel schema

Keep one JSONL file per run. Every line is an event with a kind, a span id, an optional parent, and a payload that can be hashed after sorting keys. The point is joinability. A log line without a span id is orphan evidence. A tool call without a parent is an unrooted commit. Both should fail the capture before you even diff.

# proposal: local capture, not a vendor SDK
from __future__ import annotations

import hashlib, json, time
from dataclasses import dataclass, field
from typing import Any, Iterable

ALLOWED_KINDS = {"span_open", "span_close", "tool", "log"}

@dataclass
class Event:
    kind: str
    span_id: str
    parent_id: str | None
    name: str
    attrs: dict[str, Any] = field(default_factory=dict)
    ts_mono: float = field(default_factory=time.monotonic)

    def validate(self) -> None:
        if self.kind not in ALLOWED_KINDS:
            raise ValueError(f"unknown kind: {self.kind}")
        if not self.span_id:
            raise ValueError("span_id required")
        if self.kind == "log" and self.parent_id is None and not self.span_id:
            raise ValueError("log must join a span")

def _stable(obj: Any) -> str:
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), default=str)

def arg_digest(attrs: dict[str, Any], redact: Iterable[str] = ("api_key", "token", "password")) -> str:
    cleaned = {k: ("<redacted>" if k.lower() in redact else v) for k, v in attrs.items()}
    return hashlib.sha256(_stable(cleaned).encode()).hexdigest()[:16]

def fingerprint(ev: Event) -> tuple:
    # monotonic time is recorded, never used as order
    return (ev.kind, ev.name, ev.span_id, ev.parent_id or "", arg_digest(ev.attrs))

Record in causal order as the loop actually calls tools. Do not sort by ts_mono later and pretend that is history. If a worker interleaves two agent runs, give each run its own file. Mixing forests in one stream is how a “fix” absorbs a neighbor’s spans.

class Capture:
    def __init__(self, run_id: str) -> None:
        self.run_id = run_id
        self.events: list[Event] = []
        self._open: set[str] = set()

    def open_span(self, span_id: str, name: str, parent_id: str | None = None, **attrs: Any) -> None:
        ev = Event("span_open", span_id, parent_id, name, attrs)
        ev.validate()
        self._open.add(span_id)
        self.events.append(ev)

    def tool(self, span_id: str, name: str, parent_id: str, **attrs: Any) -> None:
        if parent_id not in self._open:
            raise ValueError(f"tool {name} cited closed or missing parent {parent_id}")
        ev = Event("tool", span_id, parent_id, name, attrs)
        ev.validate()
        self.events.append(ev)

    def log(self, span_id: str, msg: str, **attrs: Any) -> None:
        if span_id not in self._open:
            raise ValueError(f"orphan log: {msg!r}")
        payload = {"msg": msg, **attrs}
        self.events.append(Event("log", span_id, span_id, "log", payload))

    def close_span(self, span_id: str) -> None:
        if span_id not in self._open:
            raise ValueError(f"double close: {span_id}")
        self._open.remove(span_id)
        self.events.append(Event("span_close", span_id, None, "close", {}))

    def finish(self) -> None:
        if self._open:
            raise ValueError(f"unclosed spans: {sorted(self._open)}")

The finish() gate is intentional. An agent that returns a string while spans remain open has not completed the unit of work you are about to diff. Close first. Diff second. That ordering keeps the second run from looking “better” because it simply dropped cleanup.

Digest, then classify the delta

Hashing the whole JSONL is too brittle. A single log message change would fail every rerun. Diff at the grain you can explain: tool names in order, argument digests, span names with parent edges, and a count of joinable logs. Everything else is noise until you promote it.

from collections import Counter

@dataclass(frozen=True)
class Digest:
    tools: tuple[tuple[str, str], ...]          # (name, arg_digest)
    edges: tuple[tuple[str, str], ...]          # (parent_name, child_name)
    log_join: int
    kinds: tuple[tuple[str, int], ...]

def digest(cap: Capture) -> Digest:
    tools = []
    names = {}
    edges = []
    logs = 0
    for ev in cap.events:
        if ev.kind == "span_open":
            names[ev.span_id] = ev.name
            if ev.parent_id:
                edges.append((names.get(ev.parent_id, ev.parent_id), ev.name))
        elif ev.kind == "tool":
            tools.append((ev.name, arg_digest(ev.attrs)))
        elif ev.kind == "log":
            logs += 1
    kind_counts = tuple(sorted(Counter(ev.kind for ev in cap.events).items()))
    return Digest(tuple(tools), tuple(edges), logs, kind_counts)

@dataclass
class Delta:
    added_tools: list[tuple[str, str]]
    dropped_tools: list[tuple[str, str]]
    arg_drift: list[str]
    edge_churn: list[tuple[str, str]]
    log_delta: int

def diff_traces(before: Digest, after: Digest) -> Delta:
    b_tools, a_tools = list(before.tools), list(after.tools)
    added = [t for t in a_tools if t not in b_tools]
    dropped = [t for t in b_tools if t not in a_tools]
    # same tool name, different digest, same position-ish
    drift = []
    for (n1, d1), (n2, d2) in zip(b_tools, a_tools):
        if n1 == n2 and d1 != d2:
            drift.append(n1)
    churn = [e for e in after.edges if e not in before.edges]
    return Delta(added, dropped, drift, churn, after.log_join - before.log_join)

Call a rerun bounded when every class of change is either empty or explicitly allowed. Added tools are the usual surprise. Argument drift is the quiet one: the model still calls search, but the query picked up a hallucinated filter. Edge churn means the forest rewired even if the tool list looks familiar. Log delta is a weak signal; use it as a smell, not as proof.

ALLOW_ADDED = {("cache_lookup",)}
ALLOW_DRIFT = {"search"}  # only if you stubbed the corpus

def gate_fix(delta: Delta) -> None:
    unexpected_added = [t for t in delta.added_tools if (t[0],) not in ALLOW_ADDED and t[0] not in {a[0] for a in ALLOW_ADDED}]
    if unexpected_added:
        raise AssertionError(f"unbounded add: {unexpected_added}")
    if delta.dropped_tools:
        raise AssertionError(f"dropped tools: {delta.dropped_tools}")
    bad_drift = [n for n in delta.arg_drift if n not in ALLOW_DRIFT]
    if bad_drift:
        raise AssertionError(f"arg drift: {bad_drift}")
    if delta.edge_churn:
        raise AssertionError(f"parent rewrite: {delta.edge_churn}")

Wire that gate after the second capture, not after a human reads stdout. A CI job can store both JSONL files as artifacts. The assertion message is the debug entry point: which tool appeared, which argument hash moved, which parent edge is new. You then open those two files and read only the unmatched events. That is the reusable loop. It does not require a dashboard.

Where a free model path belongs

The expensive part of this method is not the diff. It is obtaining two captures under comparable constraints: same stubbed tools, same redaction, same close-span rule. If inference cost contaminates the experiment, people skip the second capture and declare victory from a single pretty answer.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option. Those two facts are the only product claims here. They matter only as a way to run the failing capture and the candidate capture without folding billed tokens into the comparison. They do not replace stubs, redaction, or the gate above, and they do not imply a particular model name, quota, or hardware profile.

If you already have a stable paid endpoint, keep using it. The oracle is the bounded diff, not the vendor. A free path is useful when the alternative is not measuring the second run at all.

A worked walk through one failure class

Suppose run A calls retrieve then summarize. Run B, after a prompt tweak, calls retrieve, retrieve, then summarize. Final text can look cleaner because the second retrieve stuffed the context. gate_fix should fail on added_tools. The allowlist stays empty until you decide duplicate retrieve is the intended contract. That decision belongs in code, not in a chat thread.

Argument drift is the next class. Run A hashes {"q": "span cardinality"}. Run B hashes {"q": "span cardinality 2024"}. Same tool name, different digest. If your corpus is stubbed, you may allow search drift while you iterate on phrasing. If the tool hits a live index, do not allow it. Live indexes turn a prompt change into an uncontrolled input, and the diff will bounce forever.

Orphan logs are a capture bug, not an agent bug. A print inside a tool that forgot the span id will trip Capture.log. Fix the logger before you debate the model. Otherwise the digest’s log_join count becomes a random integer, and you will start allowing log delta because it is noisy. That is how the oracle decays.

Limitations

This recorder is not OpenTelemetry, not a distributed context propagator, and not a substitute for evaluating answer quality. Hashing sorted JSON will still thrash if tool adapters inject timestamps or request ids into arguments. Strip those fields before arg_digest, or every rerun looks unbounded.

Non-deterministic tools without stubs cannot use a strict tool-sequence gate. You must either freeze the tool, hash a coarser view (name only), or stop claiming the second run is comparable. Streaming token traces are out of scope. So is multi-tenant mixing: one JSONL per run_id, always.

The allowlists are load-bearing. An unbounded ALLOW_DRIFT turns the gate into theater. Review them when the prompt changes, not once per quarter. And never persist raw tool arguments that contain secrets. The digest is supposed to be shareable. The JSONL may not be.

Who should not use this

Do not adopt this loop if your agent is a single deterministic function with no tools. A unit test on the return value is enough. Do not adopt it if you cannot close spans or join logs; the capture will fail closed, which is correct, but you will not get a diff. Do not adopt it as a product tour. If the goal is to try a chat UI, you do not need a trace oracle.

Teams that ship user PII through tool arguments should not store JSONL until redaction is real. The arg_digest helper only covers obvious key names. That is not a compliance program.

The core conclusion does not move. A green rerun is a candidate. It becomes a fix when the trace diff is bounded, the added tools are intended, the arguments that drifted are allowlisted, and every log still names a live span. Until then you only have a nicer last line.