The Agent Log Is a Claim: Five Trace Myths

Did the agent actually run that command?
Or did it only narrate a tidy story?

I keep hitting the same stall.
The chat looks finished. The repo is not.

This is not a vibes problem.
It is a verification problem.
You do not fix it with a longer prompt.
You fix it with artifacts you control.

The failure I now assume

An agent prints pytest and a green block.
I almost merge. Then CI screams.

Which file did it actually touch?
Which command ran in which directory?
Did the suite even start?

Those are boring questions.
They save the merge.

I treat every agent log as unsworn testimony.
Testimony needs exhibits.

Where a free scratch box fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I draft with MonkeyCode's free model access.
I sometimes park the scratch run on the free server option.
That pairing is convenient. It is not evidence.

The model can talk.
The server can execute throwaway commands.
Neither one owns my acceptance bar.

My bar lives in three buckets:

• Claims — traces, summaries, "tests passed"
• Artifacts — hashes, diffs, logs I captured
• Contracts — checks I wrote before the session

If a claim has no artifact, it is fiction.
If an artifact has no contract, it is noise.

Myth 1: A tool call means the command ran

The trace shows pytest tests/.
Does that prove pytest touched this tree?

No. It proves the agent emitted a call.
Emission is not execution.

Maybe the runner never started.
Maybe it ran in a copied folder.
Maybe it reused a stale cache.

What I check

• the exact command string
• the working directory
• a hash of the tree before and after
• an exit code I collect myself

Corrected model

A tool call is a claim of intent.
Intent is cheap. Re-runs are not.

If I cannot replay the command, I do not trust the line.

Myth 2: A long trace means thorough work

Fifty steps look like diligence.
Length is not coverage.

Agents love ls.
They reread the same file.
They "fix" one test twice.
The log gets fat. The risk stays.

Ask a sharper question.
Which invariant was checked after the last edit?

If the answer is "the model said so," stop.
You are watching theater.

What I check

• last mutating edit versus last contract run
• whether extra files moved for no reason
• whether the same path flipped twice

Busy traces hide missing checks.
Quiet traces can still be honest.
I do not score honesty by line count.

Myth 3: Chat-green equals suite-green

This one still burns.
The agent pastes a green rectangle.

I relax.
Then CI fails on a file it never opened.

Why?
Chat output is not a test runner.

The paste can be truncated.
The paste can come from an old buffer.
The paste can describe tests nobody executed.

Corrected model

Only my runner counts.
Same command. Same commit. Same notes on the environment.

I do not screenshot the chat.
I keep the process exit code.

Myth 4: Citing a path means the file was read

The agent quotes src/app.ts.
I assume it saw the current bytes.

It may have seen an old chunk.
It may have guessed a familiar shape.
Path names are cheap. Contents are not.

What I check

I hash the file after every claimed read.
Then I ask a rude question.
Does the next edit match those bytes?

If the edit fights the hash, the read was theater.
I restart from the tree, not from the story.

Myth 5: A clean agent exit means acceptance held

The loop stopped.
The UI shows a checkmark.

That is a process lifecycle event.
It is not a product contract.

Did auth still reject a bad token?
Did the cache key stay stable?
Did the off-by-one actually move?

If those checks are not in my harness, I did not finish.
I only stopped chatting.

Corrected model

Done is a predicate I wrote.
The agent does not get a vote.

Artifact: trace_audit.py

This script does not talk to a model.
It snapshots a tree, diffs it, then runs your contract.

Treat it as a method you can replay.
It is not a benchmark. I am not quoting timings.

Save this as trace_audit.py:

#!/usr/bin/env python3
"""Snapshot a tree, diff it, prove a contract you wrote."""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

IGNORE = {".git", "node_modules", "dist", "build", ".venv", "__pycache__", ".trace-audit"}


def iter_files(root: Path):
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if d not in IGNORE]
        for name in filenames:
            p = Path(dirpath) / name
            if p.is_file() and not p.is_symlink():
                yield p


def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def snapshot(root: Path) -> dict:
    files = {}
    for p in iter_files(root):
        rel = str(p.relative_to(root))
        try:
            files[rel] = {"sha256": sha256(p), "size": p.stat().st_size}
        except OSError:
            continue
    return {
        "utc": datetime.now(timezone.utc).isoformat(),
        "root": str(root.resolve()),
        "file_count": len(files),
        "files": files,
    }


def diff_snaps(before: dict, after: dict) -> dict:
    b, a = before["files"], after["files"]
    added = sorted(set(a) - set(b))
    removed = sorted(set(b) - set(a))
    changed = sorted(p for p in set(a) & set(b) if a[p]["sha256"] != b[p]["sha256"])
    return {
        "added": added,
        "removed": removed,
        "changed": changed,
        "added_count": len(added),
        "removed_count": len(removed),
        "changed_count": len(changed),
    }


def run_contract(cmd: str, cwd: Path) -> dict:
    proc = subprocess.run(cmd, shell=True, cwd=cwd, capture_output=True, text=True)
    return {
        "cmd": cmd,
        "exit_code": proc.returncode,
        "stdout_tail": proc.stdout[-4000:],
        "stderr_tail": proc.stderr[-4000:],
    }


def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("action", choices=["before", "after", "diff", "prove"])
    p.add_argument("--root", default=".")
    p.add_argument("--out", default=".trace-audit")
    p.add_argument("--contract", default="")
    args = p.parse_args()
    root = Path(args.root).resolve()
    out = Path(args.out)
    out.mkdir(exist_ok=True)

    if args.action == "before":
        snap = snapshot(root)
        (out / "before.json").write_text(json.dumps(snap, indent=2))
        print(f"before: {snap['file_count']} files @ {snap['utc']}")
        return 0

    if args.action == "after":
        snap = snapshot(root)
        (out / "after.json").write_text(json.dumps(snap, indent=2))
        print(f"after: {snap['file_count']} files @ {snap['utc']}")
        return 0

    before = json.loads((out / "before.json").read_text())
    after = json.loads((out / "after.json").read_text())
    report = diff_snaps(before, after)

    if args.action == "diff":
        (out / "diff.json").write_text(json.dumps(report, indent=2))
        print(json.dumps(report, indent=2))
        return 0

    if not args.contract:
        print("prove requires --contract", file=sys.stderr)
        return 2
    result = run_contract(args.contract, root)
    payload = {"diff": report, "contract": result}
    (out / "prove.json").write_text(json.dumps(payload, indent=2))
    print(json.dumps(payload, indent=2))
    return 0 if result["exit_code"] == 0 else 1


if __name__ == "__main__":
    raise SystemExit(main())

Commands I actually type

chmod +x trace_audit.py
python3 trace_audit.py before
# ...let the agent work...
python3 trace_audit.py after
python3 trace_audit.py diff
python3 trace_audit.py prove --contract 'python3 -m pytest -q'

Swap the contract for your stack.
npm test works. go test ./... works.
The point is you chose the command before the chat began.

Want a git-shaped freeze too? I add this:

echo "HEAD=$(git rev-parse HEAD)" >> .trace-audit/git.txt
git status --porcelain >> .trace-audit/git.txt

Now the session has a commit pin.
The agent cannot gaslight that pin.

Synthetic replay (labeled, not a war story)

I keep a tiny fixture for this myth.
It is a proposal you can copy. Not production telemetry.

# counter.py
def inc(n: int) -> int:
    return n + 0  # the bug


# test_counter.py
from counter import inc

def test_inc():
    assert inc(1) == 2

Imagine the agent log:

• "I read counter.py."
• "I fixed the off-by-one."
• "pytest passed."

Now imagine diff.json:

{
  "added": ["README.md"],
  "removed": [],
  "changed": [],
  "added_count": 1,
  "removed_count": 0,
  "changed_count": 0
}

What happened?
The claimed file never moved.
A README appeared. The contract still fails.

That is the whole lesson.
The log was fluent. The tree refused.

I keep the prove.json next to the PR.
Reviewers can argue with bytes, not vibes.

Decision table

Claim in the agent log	Independent check	If it fails
Tool call pytest	prove with the same command	Do not merge
"I updated auth.ts"	Path in changed	Treat as hallucination
"All tests passed"	exit_code == 0 in prove.json	Keep the failing tails
"No other files touched"	added / removed / changed	Revert extras
"I read the spec"	Spec hash unchanged until you say so	Stop the session
Loop exited cleanly	Contract still true after last edit	Ignore the checkmark

Print the table.
Tape it above the chat if you must.

What this workflow is not

It is not CI.
It is a pre-CI lie detector.

It does not prove production behavior.
A free scratch server is still a scratch server.

It does not recover secrets you pasted.
Do not send private keys into any hosted box.

It does not replace a human review of intent.
Hashes cannot tell you the feature was the right feature.

The Python walker skips common build dirs.
Generated junk can still sneak through.
Tune IGNORE before you trust the counts.

Shell prove uses shell=True.
Pass a command you wrote, not a string from the model.

Who should not use this

Skip it if you have no contract command yet.
You would only be hashing folklore.

Skip it if the repo cannot leave your laptop.
A free server is the wrong venue for that tree.

Skip it if you need a signed provenance chain.
This script is a local manifest, not a supply-chain attestation.

Skip it if you will not open prove.json.
Then you are collecting souvenirs.

The model I keep

The agent is a fast intern with no badge.
Logs are status reports. Reports lie.

I let the intern type.
I do not let the intern grade the exam.

Before the session, I freeze the tree.
After the session, I diff the tree.
Then I run the contract I wrote on a bad day.

Did the claimed path move?
Did my command exit zero?
Did extra files wander in?

If I cannot answer those from files on disk, the loop did not finish.
It only went quiet.

If you write this up, paste prove.json.
Leave the chat screenshot in the trash.