Reconcile Logs, Tool Calls, and Diffs Before You Trust a Trace

An agent trace is not one record. It is three records that never fully agree: the log line, the tool call, and the diff. The log says what the program believed was happening. The tool record says what the runtime actually dispatched. The diff says what the filesystem finally accepted.

When those three disagree, you get a bug class that no single source can show you. A green log with a red diff is an unlogged write. A dispatched call with an empty diff is a lie about work done. A log that announces a retry the runtime never issued is an incident note that will mislead the next reader. Reconcile the three before you read anything else in the trace.

The fixture first, because a harness you cannot reproduce is a rumor

Start with a trace you can hand to a colleague. This one has a clean read, a clean write, and two planted disagreements.

{
  "run_id": "r-42",
  "logs": [
    {"t": 0.00, "msg": "plan ready"},
    {"t": 0.40, "msg": "tool read_file start"},
    {"t": 0.90, "msg": "tool read_file ok"},
    {"t": 1.20, "msg": "retrying write_file"}
  ],
  "tool_calls": [
    {"name": "read_file", "start": 0.40, "end": 0.90, "args": {"path": "src/app.py"}},
    {"name": "write_file", "start": 1.10, "end": 1.60, "args": {"path": "src/app.py"}}
  ],
  "diffs": [
    {"path": "src/app.py", "added": 4, "removed": 0},
    {"path": "tests/test_app.py", "added": 2, "removed": 1}
  ]
}

Read it as a reviewer and you already feel the friction. Something edited a test file that no tool call claims to have touched. Something said "retrying" while only one write exists. The point of the harness is to make that friction mechanical, so a nightly job catches it instead of a tired human at 11pm.

A harness small enough to audit

The join is plain set arithmetic plus one regex, and it returns a list rather than printing as it goes, which keeps it testable.

#!/usr/bin/env python3
"""reconcile.py - join logs, tool calls, and diffs into one finding list."""
import json
import re
import sys

MUTATING = {"write_file", "edit_file", "delete_file", "apply_patch"}
BLOCKING = {"silent_write", "phantom_call"}


def reconcile(trace):
    findings = []
    calls = [c for c in trace["tool_calls"] if c["name"] in MUTATING]
    written = {c["args"].get("path"): c for c in calls}
    diff_paths = {d["path"]: d for d in trace["diffs"]}

    for path, d in diff_paths.items():
        if path not in written:
            findings.append({"kind": "silent_write", "path": path,
                             "detail": f'+{d["added"]}/-{d["removed"]}, no mutating call'})

    for path, c in written.items():
        if path not in diff_paths:
            findings.append({"kind": "phantom_call", "path": path,
                             "detail": f'{c["name"]} dispatched, nothing changed'})

    for line in trace["logs"]:
        m = re.search(r"retry\w*\s+(\w+)", line["msg"])
        if not m:
            continue
        name = m.group(1)
        seen = sum(1 for c in trace["tool_calls"] if c["name"] == name)
        if seen < 2:
            findings.append({"kind": "retry_unrecorded", "path": name,
                             "detail": f'log@{line["t"]:.2f} claims retry of {name}; {seen} call recorded'})
    return findings


def main(path):
    findings = reconcile(json.load(open(path)))
    for f in findings:
        print(f'{f["kind"]:<16} {f["path"]:<20} {f["detail"]}')
    blocking = [f for f in findings if f["kind"] in BLOCKING]
    print(f'{len(findings)} findings ({len(blocking)} blocking)')
    return 1 if blocking else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it against the fixture and the disagreements surface in a fixed order, which matters if you plan to diff the output between runs.

$ python reconcile.py trace.json
silent_write     tests/test_app.py    +2/-1, no mutating call
retry_unrecorded write_file           log@1.20 claims retry of write_file; 1 call recorded
2 findings (1 blocking)
$ echo $?
1

The exit code is the contract. Advisory findings (an unexplained retry) return 0 and inform. Blocking findings (a mutation nobody dispatched) return 1 and stop the pipeline. That split is the whole design: you want the retry note in the report, and you want the silent write to fail the run.

Four disagreements worth naming

Kind	Shape of the evidence	Blocking
silent_write	Diff path with no mutating call	Yes
phantom_call	Mutating call with no diff at its declared path	Yes
retry_unrecorded	Log claims a retry the call list cannot confirm	No
redirect_suspect	Exactly one unexplained write and one unexplained call, paired by order	Review

The last row is a heuristic and should stay labelled as one. Pairing a lone orphan call with a lone orphan diff is a reasonable guess, not proof. Keep it advisory until you have seen it be right on your own traces.

What this cannot see

Read-only tools are exempt by the MUTATING set, so a shell tool that mutates files through sed -i will produce a false silent_write. Path normalization bites too: symlinks, absolute versus relative paths, and case differences all break the join. Concurrent runs sharing a working tree will interleave diffs and produce findings that are artifacts of scheduling rather than agent behavior.

In-memory state, network side effects, and anything that never touches the working tree are simply outside the model. The harness answers one narrow question — do these three records describe the same run — and it answers it only as well as your tool registry is honest.

The fixture test comes next, because a harness with no test is just another opinion. Ten lines of pytest pinning the exact finding set is enough.

from reconcile import reconcile
import json


def test_fixture_findings():
    trace = json.load(open("trace.json"))
    kinds = {(f["kind"], f["path"]) for f in reconcile(trace)}
    assert kinds == {("silent_write", "tests/test_app.py"),
                     ("retry_unrecorded", "write_file")}

Running it where the traces already land

A reconciliation step earns its keep as a scheduled job, not a manual script. Once a night, over yesterday's traces, writing findings to disk and letting the exit code drive a notification is the smallest useful version. I run that nightly job on the free server option of MonkeyCode rather than on a laptop, so the harness sits next to the trace store instead of depending on someone remembering to run it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free-tier terms and limits change, so check the project's own page for the current offer rather than trusting any number quoted in a blog post.

Free model access is a good fit for exactly one narrow job here: mapping tool names from different SDKs (str_replace_editor, apply_patch, write) onto your MUTATING taxonomy. That mapping is boring, it breaks whenever you add an integration, and it must never be trusted blindly. Sketch, not production code:

# pseudocode: model proposes a mapping, the frozen fixture verifies it
proposed = classify_tool_names(unmapped_names)          # model call
assert proposed == FROZEN_EXPECTED                      # test gate
TAXONOMY.update(proposed if proposed else RULE_MAP)     # rule map fallback

The rule map is the default. The model fills gaps between releases, and the frozen fixture decides whether the answer ships. If the assertion fails, you fall back and read the diff — which is the same discipline the harness itself is built on.

Who should skip this

Single-turn scripts with no file mutations will produce empty finding lists forever. Teams already capturing full OpenTelemetry spans plus a diff-per-tool-call record have solved this at the source, and a reconciliation pass on top is redundant ceremony. And if your traces are large enough that a nightly Python pass over them is the expensive part, fix retention first — reconciliation over a sampled week usually tells you everything a full month would.

If you keep a trace store and have no reconciliation step, run this against your last twenty runs before you change any prompt. The disagreements you find tend to be older than your current bug.