Your Agent Said "Done". The Trace Shows Nothing Changed.

Picture a routine cleanup run on a staging repo.

Fourteen tool calls. Fourteen status: ok spans. Then the post-run diff is empty.

Nothing was deleted. Nothing was renamed. The agent still reported success.

This is the failure class that traces miss most often. The call succeeded. The world did not change.

The bug is in your status field, not your model

Most trace schemas record three things: tool name, arguments, output. Status comes from the caller's try/except.

That tells you the function returned. It does not tell you the effect happened.

• A prune_branches call returns ok when zero branches match the filter.
• A migrate_rows call returns ok when the batch was already migrated.
• A write_config call returns ok when it rewrote identical bytes.

All three are legitimate no-ops. All three look identical to real work in a trace.

When an agent loops, retries, or replans, this ambiguity compounds fast. You cannot tell "it worked" from "it politely did nothing."

Add a witness: probe the state you mutate

You do not need distributed tracing to fix this. You need one cheap, read-only state probe per mutating tool.

A probe returns a small snapshot. Hash it. Compare before and after. Store the digest pair in the span.

# witness.py — record a state probe before and after every mutating tool call
import hashlib
import json
import time
import uuid


def digest(value) -> str:
    raw = json.dumps(value, sort_keys=True, default=str).encode()
    return hashlib.sha256(raw).hexdigest()[:12]


def record_effect(trace_path, tool_name, args, call, probe):
    """probe: zero-arg callable returning a small, read-only state snapshot."""
    before = digest(probe())
    span_id = uuid.uuid4().hex[:8]
    status, output = "ok", None

    t0 = time.monotonic()
    try:
        output = call(**args)
    except Exception as exc:          # keep the span even when the tool explodes
        status, output = "error", {"error": repr(exc)}
    elapsed_ms = round((time.monotonic() - t0) * 1000, 2)

    after = digest(probe())
    span = {
        "span_id": span_id,
        "tool": tool_name,
        "args_digest": digest(args),
        "status": status,
        "elapsed_ms": elapsed_ms,
        "effect": {"before": before, "after": after, "changed": before != after},
        "output_digest": digest(output),
    }
    with open(trace_path, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(span) + "\n")
    return output

Wiring it up is one line per tool:

probe = lambda: {
    "branches": sorted(repo.branches()),
    "head": repo.head_sha(),
}

record_effect(
    "trace.jsonl",
    "prune_branches",
    {"older_than_days": 90},
    repo.prune,
    probe,
)

Good probes are boring. Three rules keep them honest:

1. Read-only. A probe must never mutate.
2. Bounded. Return counts, ids, hashes — not full tables.
3. Stable. Sort collections so ordering noise does not fake a change.

Avoid updated_at timestamps in the probe. They burn the whole trick by changing on every touch.

Four cells, four different bugs

The value shows up when you cross status with the effect flag. You get a 2x2 that separates failure modes you were previously merging.

status	effect changed	label	what it usually means
ok	yes	clean_success	expected; keep moving
ok	no	silent_success	no-op, over-broad filter, or wrong target
error	yes	phantom_failure	partial write, timeout after commit, retry hazard
error	no	clean_failure	expected; safe to retry

silent_success is the cell that wastes the most engineer hours. The agent believes it made progress. It replans, retries, and burns context on a task that finished zero work.

phantom_failure is the dangerous cell. Something changed, but the caller thinks it failed. Any retry now risks a duplicate effect.

The checker: fail the run, not the review

A trace is only useful if a script can read it. Keep the classifier in its own file and exit non-zero.

# check_effects.py — classify a JSONL trace and fail CI on suspicion
import collections
import json
import sys


def classify(span):
    if span.get("effect") is None:
        return "unprobed"
    ok = span["status"] == "ok"
    changed = span["effect"]["changed"]
    if ok and changed:
        return "clean_success"
    if ok and not changed:
        return "silent_success"
    if changed:
        return "phantom_failure"
    return "clean_failure"


rows = [json.loads(line) for line in open(sys.argv[1], encoding="utf-8")]
counts = collections.Counter(classify(s) for s in rows)

for name, n in counts.most_common():
    print(f"{n:>4}  {name}")

suspicious = [s for s in rows if classify(s) in ("silent_success", "phantom_failure")]
print("\nSuspicious spans:")
for s in suspicious:
    print(f"  {s['span_id']}  {s['tool']:<24} {classify(s):<16} {s['elapsed_ms']}ms")

sys.exit(1 if suspicious else 0)

Run it the same way every time:

python check_effects.py trace.jsonl

Now a silent no-op is a red build, not a mystery in a UI timeline.

A debug loop you can reuse

This is the part worth writing down. Same order, every time.

1. Re-run the failing task with witnesses enabled.
2. Read the counter output before reading any span.
3. Open the first silent_success span, not the last failure.
4. Check whether the filter matched nothing, or matched the wrong object.
5. Fix the argument selection, then re-run the identical task.
6. Confirm the same span flipped to clean_success.
7. Only then look at whether the agent's final answer changed.

Step 3 is the counterintuitive one. The first no-op is almost always the root cause. Later spans are downstream noise from a confused planner.

Step 5 matters for reproducibility. Change one thing per run. A trace diff with five variables is not evidence.

Where a free model and a free server actually help

Two parts of this loop are a good fit for cheap infrastructure, and one of them is boring enough that people skip it.

First: writing probes. Every tool schema already describes the state it touches. Asking a model to draft a candidate probe from a schema is a small, one-shot job. Bad drafts are obvious and cost nothing to throw away. This is where free model access is useful — the task is small, repeatable, and you review every output anyway.

Second: running the loop. You need a box that holds trace.jsonl, runs check_effects.py, and serves a page of span rows. That is not production. It is a scratch environment you will rebuild. A free server option removes the provisioning step, which is exactly the step that kills these experiments before the second iteration.

MonkeyCode offers an open-source project with free model access and a free server option, which is why it came up here. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free tiers as an accelerator, not an architecture. Check the project page for current limits and terms before you build a workflow on them. Anything that must be permanent belongs in your own infrastructure.

Where the witness approach breaks down

Be honest about the edges before you adopt this.

• Probes cost time. Every mutating call now does two extra reads. On a hot path, that is measurable.
• Non-idempotent probes lie. A probe that consumes a queue message destroys the thing it measures.
• "Changed" is not "correct." A span can flip to clean_success while writing the wrong value. The witness proves motion, not intent.
• Large state defeats the digest. Hashing a million rows teaches you nothing useful.
• Probes can leak secrets into your trace store. Digest them, or redact the snapshot before writing.
• External systems may lag. An eventually consistent read makes silent_success a false alarm. Wait, or probe a source that is immediate.

Skip this approach if your agent is purely read-only. Skip it if the tools expose no cheap state. Skip it if your team needs sub-millisecond overhead and cannot afford a second read.

The takeaway

A span that says ok documents a return value. A span that says ok plus a changed digest documents work.

You only need one of those when the agent insists the job is done. Add the probe, cross the two columns, and let the counter tell you which spans are lying.

If you want to try the loop without provisioning anything, the project's free model access and free server option are the shortest path to your first silent_success count.