An agent's end-of-run summary is a claim about work done, and the trace is the only artifact that can support or break it. This article is a claim-audit loop: a small script that replays a run's tool-call log, builds a fact ledger from calls that actually succeeded, and marks every summary claim as PASS, UNSUPPORTED, or CONTRADICTED. The loop runs on a free model endpoint and a free server, and at a 10 million token allowance it funds roughly 2,500 checkpoint re-runs per month before a single dollar changes hands.
The strongest DEV discussions of the past week circled the same gap from different angles. What is the human supposed to do while the model writes the code? How do you audit a reviewer that is itself a model? Why does a model trust every fact in its context window? All three reduce to one missing piece: nobody audits the final report against the actual run. A reviewer reads the summary, and a summary compresses ten minutes of tool calls into three sentences. Compression is where failures go to hide.
Consider the classic green-check failure. The agent reads the config, patches it, writes it, restarts the service, and reports "done." If the first write failed validation and the agent retried the same write with a cleared error flag, the summary still says done. The trace shows the fail-then-retry sequence; the summary shows a checkmark. The reviewer never sees that the checkmark sits on a swallowed error.
The fix is mechanical: log every tool call as a JSONL event, then audit the summary against the ledger. This is the trace format the loop expects:
{"ts": 1785596400.12, "type": "tool_call", "tool": "config.write", "ok": false, "error": "validation failed: retry_interval out of range"}
{"ts": 1785596401.40, "type": "tool_call", "tool": "config.write", "ok": true}
{"ts": 1785596402.11, "type": "tool_call", "tool": "service.restart", "ok": true}
{"ts": 1785596402.87, "type": "tool_call", "tool": "health.check", "ok": true}
A small wrapper around your tool runner produces the ledger without touching the agent's logic:
import json, time
def traced(fn, tool_name):
def wrapper(**kw):
record = {"ts": time.time(), "type": "tool_call", "tool": tool_name}
try:
result = fn(**kw)
record["ok"] = True
except Exception as exc:
record["ok"] = False
record["error"] = str(exc)
print(json.dumps(record))
raise
print(json.dumps(record))
return result
return wrapper
The auditor replays the ledger against the claims the agent made. A claim is a statement plus the tool calls that must have succeeded to support it:
[
{"claim": "updated nginx retry policy",
"must_show": ["config.read", "config.write"],
"must_not_fail": ["config.write"]},
{"claim": "restarted the service",
"must_show": ["service.restart", "health.check"]}
]
And the audit script itself:
import json, sys
from collections import Counter
def replay(path):
ledger = []
for line in open(path):
if not line.strip():
continue
ev = json.loads(line)
if ev.get("type") == "tool_call":
ledger.append(ev)
return ledger
def audit(ledger, claims):
ok = Counter(e["tool"] for e in ledger if e["ok"])
failed = Counter(e["tool"] for e in ledger if not e["ok"])
for c in claims:
supported = all(ok.get(t, 0) > 0 for t in c["must_show"])
contradicted = any(failed.get(t, 0) > 0 for t in c.get("must_not_fail", []))
verdict = "PASS" if supported and not contradicted else \
"CONTRADICTED" if contradicted else "UNSUPPORTED"
print(f"{verdict:12} {c['claim']}")
if verdict != "PASS":
print(f" ok={dict(ok)} failed={dict(failed)}")
if __name__ == "__main__":
ledger = replay(sys.argv[1])
claims = json.load(open(sys.argv[2]))
audit(ledger, claims)
Point it at the trace above, and it prints the truth the summary hid:
CONTRADICTED updated nginx retry policy
ok={'config.write': 1, 'service.restart': 1, 'health.check': 1} failed={'config.write': 1}
PASS restarted the service
Note what the auditor checks: the existence of successful calls, never the content of the writes. It will not catch a config that validates but remains semantically wrong. It will catch the retry-after-failure pattern every time, because the failed call stays in the ledger.
The debug loop follows from the verdict. When a claim comes back CONTRADICTED, do not re-run the agent from zero. Find the last PASS claim in the ledger, cut the context right after its final supporting call, and re-run only the failed segment with a narrower instruction. That is checkpoint-restart debugging, and it turns one vague "fix the deployment" ticket into a bounded re-run that costs a few thousand tokens instead of a whole fresh run.
Where you run it matters less than that it runs. For an unattended loop you need model access for the re-run segments plus a server to hold the scheduler, and both can be free. MonkeyCode's free model access covers the first, its free server option covers the second, and because the project is open source, the deployment path is inspectable rather than assumed. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The budget math is the honest part. If one checkpoint re-run costs 4,000 tokens on average — the trace segment, the instruction, the completion — a 10 million token allowance buys about 2,500 audits. The formula is allowance / average_cost_per_rerun, and the ledger gives you the real average from your own runs instead of a guess.
Who should not use this loop? Teams that already run distributed tracing with alerting will find a local script redundant. Solo developers with ten-step runs should audit by eye. And a free server has no uptime contract, so a production SLO on donated infrastructure is a contradiction in terms. The loop is a debugging practice, not a monitoring product.
The pattern to steal is not the script; it is the default. Treat every summary as a hypothesis and the trace as the test. If your agent stack is young and you want this running by the weekend, MonkeyCode's free model access and free server option are a low-friction place to start. Copy the script, adapt the event format, and let the ledger disagree with the report out loud.
Top comments (0)