Diff the Run Card, Not the Log Stream

A single agent log is a diary. It is not a diagnosis. The object that actually debugs a tool-using run is a compact run card, compared against a prior card the way git diff compares trees: four channels, one verdict, no scrolling.

Most agent failures do not announce themselves as exceptions. They announce themselves as drift. A tool is skipped. A file is rewritten instead of patched. A second model turn invents a constraint the first turn never stated. If you only tail stdout, those events look like more text. If you diff structured cards, they look like a failing test.

This article is a proposed local harness, not a production case study. The code is meant to be copied into a repo and pointed at any OpenAI-compatible chat endpoint you already use. Treat the JSON fixtures as examples. They are not measurements from a live fleet.

Volume is not causality

Log volume grows with retries, streaming tokens, and helper prints. Causality does not. A 4,000-line capture can hide a one-line mistake: the agent called read_file twice, never called apply_patch, and still emitted done. Humans compensate by scrolling. Machines cannot.

The better analogy is source control. Nobody reviews a commit by reading every byte of the working tree. They review the diff against a parent. Agent runs need the same parent. Without it, you are reading an orphan commit and calling it observability.

Non-determinism makes the parent more important, not less. Two runs can both report success and still diverge in tool order, path set, or the hash of files they touched. Success is a weak signal. Channel agreement is a stronger one. Weak signals waste retries. Strong ones tell you which retry is even worth running.

Four channels, one card

Keep four channels, and refuse to let any one of them stand in for the others. Logs remain, but only as a stable digest: logger name, level, event, optional error class. You do not store the novel. You store the plot points.

Tool calls are a sequence of names, argument fingerprints, and exit status. Filesystem change is a path list plus a content hash, or the output of git diff --raw when the worktree is already a git checkout. Model turns collapse to role, a SHA-256 of the content, and a character count so a runaway completion is visible without retaining the prompt body.

The card is the join of those four. The verdict is the diff of two cards. That is the whole method. If a channel is missing, the card is incomplete, and the comparison should fail closed. A green log with no tool trace is not a pass. It is an instrumentation gap wearing a success bit.

Disagreement across channels is more informative than a blanket failure. Logs say success while files stay still, and the agent narrated work it did not do. Tools move while files stay still, and the dispatcher ran but the patch did not land. Files move while tools never list a write, and something mutated the tree outside the recorded boundary. Each pattern is a different bug. Collapsing them into “the agent failed” wastes the next run.

A proposed schema

The following Python is a local, dependency-light sketch. It does not call a vendor SDK. It records what your runtime already emits, then writes a JSON card you can check in beside a fixture run. Latency is stored as evidence and dropped from identity. That split matters on a contended free endpoint, where queue delay would otherwise fail a test that should only watch side effects.

# runcard.py — proposed harness, not a framework
from __future__ import annotations

import hashlib
import json
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any


def _sha(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def _fingerprint(obj: Any) -> str:
    blob = json.dumps(obj, sort_keys=True, default=str).encode()
    return _sha(blob)


@dataclass
class LogEvent:
    logger: str
    level: str
    event: str
    err: str | None = None


@dataclass
class ToolCall:
    name: str
    args_fp: str
    ok: bool
    ms: int  # evidence only; excluded from channel identity


@dataclass
class FileTouch:
    path: str
    op: str  # add | mod | del
    content_fp: str


@dataclass
class ModelTurn:
    role: str
    content_fp: str
    chars: int


@dataclass
class RunCard:
    run_id: str
    started_at: float
    channels: dict[str, str] = field(default_factory=dict)
    logs: list[LogEvent] = field(default_factory=list)
    tools: list[ToolCall] = field(default_factory=list)
    files: list[FileTouch] = field(default_factory=list)
    turns: list[ModelTurn] = field(default_factory=list)
    notes: str = ""

    def canonical_tools(self) -> list[dict]:
        return [{"name": t.name, "args_fp": t.args_fp, "ok": t.ok} for t in self.tools]

    def seal(self) -> "RunCard":
        self.channels = {
            "logs": _fingerprint([asdict(x) for x in self.logs]),
            "tools": _fingerprint(self.canonical_tools()),
            "files": _fingerprint([asdict(x) for x in self.files]),
            "turns": _fingerprint([asdict(x) for x in self.turns]),
        }
        return self

    def dump(self, path: Path) -> None:
        path.write_text(json.dumps(asdict(self.seal()), indent=2) + "\n")

Seal the card before you write it. The four channel hashes are the index. Everything else is evidence you open when a hash moves. If you hash wall-clock duration into tools, you will debug the scheduler. That is a different incident.

Record at the boundaries you already have

Hook the recorder at four places: the logging filter, the tool dispatcher, the worktree, and the chat client. The next block is a proposed adapter. Wire it to your runtime. Do not pretend it is a product.

# recorder.py
import time
from pathlib import Path

from runcard import FileTouch, LogEvent, ModelTurn, RunCard, ToolCall, _fingerprint, _sha


class RunRecorder:
    def __init__(self, run_id: str, worktree: Path):
        self.card = RunCard(run_id=run_id, started_at=time.time())
        self.worktree = worktree
        self._before = self._snapshot()

    def log(self, logger: str, level: str, event: str, err: str | None = None) -> None:
        self.card.logs.append(LogEvent(logger, level, event, err))

    def tool(self, name: str, args: dict, ok: bool, ms: int) -> None:
        # Fingerprint an allowlisted subset if args carry timestamps.
        safe = {k: v for k, v in args.items() if k not in {"ts", "request_id"}}
        self.card.tools.append(ToolCall(name, _fingerprint(safe), ok, ms))

    def turn(self, role: str, content: str) -> None:
        blob = content.encode()
        self.card.turns.append(ModelTurn(role, _sha(blob), len(content)))

    def close(self, path: Path, notes: str = "") -> None:
        after = self._snapshot()
        self.card.files = self._touches(self._before, after)
        self.card.notes = notes
        self.card.dump(path)

    def _snapshot(self) -> dict[str, str]:
        out: dict[str, str] = {}
        if not self.worktree.exists():
            return out
        for p in self.worktree.rglob("*"):
            if p.is_file() and ".git" not in p.parts:
                out[str(p.relative_to(self.worktree))] = _sha(p.read_bytes())
        return out

    def _touches(self, before: dict[str, str], after: dict[str, str]) -> list[FileTouch]:
        files: list[FileTouch] = []
        for path in sorted(set(before) | set(after)):
            if path not in before:
                files.append(FileTouch(path, "add", after[path]))
            elif path not in after:
                files.append(FileTouch(path, "del", before[path]))
            elif before[path] != after[path]:
                files.append(FileTouch(path, "mod", after[path]))
        return files

Notice what is absent. The recorder never stores raw prompts. It stores fingerprints and lengths. Prompt bodies make cards incomparable across machines, and they turn a debug artifact into a data-boundary problem. Drop them at the edge.

A frozen capture looks like this once you have an entrypoint. Replace agent_runner with whatever already drives your loop.

mkdir -p fixtures artifacts worktree
python -m agent_runner --worktree ./worktree --out fixtures/run_base.json
python -m agent_runner --worktree ./worktree --out artifacts/run_cand.json
python diffcard.py fixtures/run_base.json artifacts/run_cand.json
echo $?

The useful CI rule is narrower than “cards must be identical”. Allow the turns hash to move when you are testing tool routing, not prose. Fail when tools or files move and you did not opt in. Token text is noisy. Side effects are not.

Diffing two cards is the test

Comparison should be boring. Named channels, an explicit allowlist, a non-zero exit when the wrong channel moves.

# diffcard.py
from __future__ import annotations

import json
import sys
from pathlib import Path

ALLOWED_DRIFT = {"turns"}  # model text may change; tools and files should not


def load(path: Path) -> dict:
    return json.loads(path.read_text())


def channel_delta(a: dict, b: dict) -> dict[str, tuple[str, str]]:
    left, right = a["channels"], b["channels"]
    moved = {}
    for name in sorted(set(left) | set(right)):
        if left.get(name) != right.get(name):
            moved[name] = (left.get(name, "missing"), right.get(name, "missing"))
    return moved


def tool_names(card: dict) -> list[str]:
    return [t["name"] for t in card.get("tools", [])]


def main(base: str, cand: str) -> int:
    a, b = load(Path(base)), load(Path(cand))
    moved = channel_delta(a, b)
    unexpected = {k: v for k, v in moved.items() if k not in ALLOWED_DRIFT}
    print(f"base={a['run_id']} cand={b['run_id']}")
    print(f"tool_seq_base={tool_names(a)}")
    print(f"tool_seq_cand={tool_names(b)}")
    print(f"moved_channels={list(moved)}")
    if unexpected:
        print(f"FAIL unexpected drift: {unexpected}")
        return 1
    print("PASS")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Put the same policy in pytest so a reviewer does not have to remember the allowlist. A card with a handful of log events, four tool calls, and two file touches is already enough to catch the classic miss: the model said it patched a file, and the worktree hash did not move.

# test_runcard_diff.py
import json
from pathlib import Path

from diffcard import channel_delta, tool_names


def test_hotfix_run_may_change_prose_not_tools():
    base = json.loads(Path("fixtures/run_base.json").read_text())
    cand = json.loads(Path("artifacts/run_cand.json").read_text())
    moved = channel_delta(base, cand)
    assert "tools" not in moved
    assert "files" not in moved
    assert tool_names(base) == tool_names(cand)

A fixture card should be small enough to read in review. The following shape is enough. Hashes are truncated here for the page, not in the real file.

{
  "run_id": "base-001",
  "channels": {
    "logs": "a3c1...",
    "tools": "91ee...",
    "files": "04b2...",
    "turns": "c778..."
  },
  "tools": [
    {"name": "read_file", "args_fp": "ab12...", "ok": true, "ms": 41},
    {"name": "apply_patch", "args_fp": "90cd...", "ok": true, "ms": 88}
  ],
  "files": [
    {"path": "src/app.py", "op": "mod", "content_fp": "e91f..."}
  ],
  "notes": "known-good hotfix path"
}

Write a one-line verdict next to the JSON. Humans read the line. Tests read the hashes.

run_id=cand-014 verdict=FAIL unexpected=files
tools=read_file,read_file,apply_patch files=src/app.py:mod
notes=assistant claimed patch applied; worktree hash unchanged on first attempt

Recent write-ups about first agent workflows keep rediscovering the same failure mode: the model fills gaps instead of calling tools. A run-card diff makes that visible without a glossary. You do not need twenty new terms. You need a parent card.

Where a free endpoint actually participates

You still need a model in the loop to generate candidate cards. A free model endpoint and a free server option are enough to exercise the harness if the workload already fits those limits. They are not a substitute for the card. They are a place the candidate run can live while fixtures and diffs stay in git.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option that can host this kind of capture loop. The method needs somewhere inexpensive to produce a second card. A hosted box does not make traces causal. If you already have another OpenAI-compatible endpoint, point the same recorder at it. The verdict file does not care which vendor hashed the turns.

Skip that path if the agent writes production data, needs a private network, or cannot tolerate a shared free-tier queue. A debug harness that depends on a contended queue will produce cards about the queue. File those under capacity, not under agent logic.

Limitations, and who should not bother

This harness does not reconstruct prompts, does not prove functional correctness, and does not replace parent-child spans. It will false-fail if tool arguments include timestamps or random request ids and you fingerprint the whole dict. Allowlist the stable keys. It will false-pass if two different patches collide in a truncated digest, so persist the full SHA-256 in any repo you actually ship. It assumes one worktree and one process. Multi-agent handoffs need a parent run_id, which this sketch does not implement.

Do not use it as a product analytics feed. Do not use it where a file snapshot would copy secrets into the card. Do not use it as your only gate if the agent is allowed to change behavior without touching files, such as a chat-only classifier. In that case the files channel is structurally empty, and you would be testing nothing while a dashboard stayed green.

The method also does not make a non-deterministic model deterministic. It makes the non-determinism visible on the channels you chose to care about. That is a smaller claim. It is the claim worth automating. Keep the base card next to the patch, and treat a moved tools hash the way you treat a moved lockfile: explain it, or revert it.