Diff the Traces, Not the Code: A Bisection Loop for Agent Runs

The AI coding shift has a quiet side effect: everyone on the team is now a reviewer. Reviewing an agent run is a bisection problem, not a reading problem. The final diff shows what changed; it rarely shows which tool call caused the change. When a run fails, the fastest diagnostic is a second trace of the same task, aligned until the paths split — the first divergence is your bug, and everything after it is downstream damage.

My previous posts covered run receipts and a trace-first debug loop; this is the step I kept missing. Receipts tell you what happened, but not where the story changed. A harness can capture every event; the hard part is deciding which one to read first. Trace differencing turns that decision into a search: run the task twice, align the streams, report the index of the first mismatch.

The schema is deliberately small. A handful of event types, one JSON object per line, nothing clever:

{"task_id":"a1b2","seq":0,"type":"model_request","prompt_tokens":812,"ts":0.12}
{"task_id":"a1b2","seq":1,"type":"tool_call","name":"read_file","path":"src/main.py","ts":0.21}
{"task_id":"a1b2","seq":2,"type":"tool_result","name":"read_file","status":"ok","ts":0.29}
{"task_id":"a1b2","seq":3,"type":"file_edit","name":"replace","path":"src/main.py","ts":0.34}

One honest note: the token field is a word count, not a real tokenizer. That precision is enough for trace comparison, and not enough for billing.

Capture is two hooks. One around the model client, one around the tool dispatcher:

import json
import sys
import time
import uuid


class TraceLogger:
    def __init__(self, sink):
        self.sink = sink
        self.seq = 0
        self.task_id = uuid.uuid4().hex[:8]

    def log(self, event_type, **fields):
        row = {
            "task_id": self.task_id,
            "seq": self.seq,
            "type": event_type,
            "ts": time.time(),
        }
        row.update(fields)
        self.seq += 1
        self.sink.write(json.dumps(row) + "\n")


trace = TraceLogger(sys.stdout)


def call_model(prompt, **kwargs):
    trace.log("model_request", prompt_tokens=len(prompt.split()))
    out = model(prompt, **kwargs)  # your client here
    trace.log("model_response", finish_reason=getattr(out, "finish_reason", None))
    return out


def run_tool(name, args):
    trace.log("tool_call", name=name, path=args.get("path"))
    result = dispatch(name, args)  # your dispatcher here
    trace.log("tool_result", name=name, status="ok" if result.ok else "error")
    return result

Then run the same task twice and diff the projections. The projection strips arguments, timestamps, and token counts; it keeps only the skeleton of the run:

import json
import sys


def load(path):
    return [json.loads(line) for line in open(path) if line.strip()]


def projection(events):
    out = []
    for e in events:
        t = e["type"]
        if t == "tool_call":
            out.append("call:" + e["name"])
        elif t == "tool_result":
            tag = e["name"]
            if e.get("status") == "error":
                tag += ":error"
            out.append("res:" + tag)
        elif t == "file_edit":
            out.append("edit:" + e["path"])
        else:
            out.append(t)
    return out


def first_divergence(g, b):
    n = min(len(g), len(b))
    for i in range(n):
        if g[i] != b[i]:
            return i
    return n


g = projection(load(sys.argv[1]))
b = projection(load(sys.argv[2]))
i = first_divergence(g, b)

print(f"common prefix: {i} events")
print(f"good: {g[i] if i < len(g) else '<end>'}")
print(f"bad : {b[i] if i < len(b) else '<end>'}")
print("-- failing run, events after the split --")
for e in load(sys.argv[2])[i:i + 6]:
    print(json.dumps(e))

Usage:

python tracediff.py good_run.jsonl bad_run.jsonl

The output answers two questions. The common-prefix length is the agent's last correct point; the first mismatched event is the inflection. Read the next six events in the failing run and you see the wrong turn with its context. More often than not, this is where agent bugs live: an early observation that was stale, mistrusted, or over-trusted. The diff finds it in seconds, before you open the final diff.

You can bisect further. Truncate the conversation at the divergence event and re-run. If the agent fails the same way, the cause is upstream; if it recovers, the divergence event itself is the cause. That rerun should be cheap, which is where free endpoints earn their place. A probe model on a free tier is enough to verify the split point; spending paid quota on regression reruns defeats the purpose of a bisection loop.

This is why I began evaluating MonkeyCode. It is an open-source project whose current offer includes free model access and a free server for hosting the harness, with a free allowance stated as 10M tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat the quota as a starting point, not a contract: free tiers change, and the 10M figure comes from the project team, not from a benchmark I ran. The README is the source of truth.

Three caveats. First, you need a baseline you trust; if both runs are garbage, a divergence is meaningless. Second, alignment is structural, not semantic: two models can solve one task in genuinely different orders, making the first difference noise. Canonicalize tool names and collapse repeated read-only calls before you diff. Third, a trace is a data leak waiting to happen — it contains prompts, file paths, and diffs. Do not push run logs to a shared server unless you control retention; a free server is convenient, not a compliance answer.

Who should skip this? Teams that never replay a failed run, and tasks with no canonical solution. If your agent writes marketing copy or one-off poems, diffing two generations tells you little. If you review twenty failing runs a week, the loop pays for itself in the first hour. The rest of the time it is a quiet habit: capture, align, bisect, fix. If you want to prototype the loop on a budget, MonkeyCode's free tier is a reasonable place to start — take two traces from your own harness on Monday and run the script. First divergence wins.