Most agent regressions are not missing instructions. They are tool arguments that never appeared in any prior observation. If a span cannot point at a source record, the call is guesswork, and guesswork should fail closed before it writes a file, opens a ticket, or spends another free-tier round trip.
Logs still help. Diffs still help. Neither answers provenance. Provenance is a cheaper question than “why did the model wander,” and it is the question most agent traces never record.
Think of a compiler that pretty-prints the build log while accepting undeclared variables. That is an agent that emits a confident tool call whose path, issue_id, or sha was never returned by list_dir, git_status, or search. The run looks complete. The trace is hollow.
This article proposes a reusable debug loop: every tool argument is tagged observed or assumed at the moment of the call, the span carries those tags, and the run fails if assumed fields cross a budget. The checker is local. It does not need a new model family, and it does not need you to believe a dashboard.
What “observed” actually means
An argument is observed when its value is byte-equal to a value stored from an earlier tool result, or when it is a strict, documented projection of that value (a filename taken from a directory listing, a commit hash taken from rev-parse). An argument is assumed when the model invented it, stitched it from the system prompt, or reused a value from a previous failed call that never landed in the ledger.
That rule is intentionally boring. Boring rules survive truncated runs. Free endpoints drop mid-stream; retry wrappers reorder stdout; parent spans lie about causality. A ledger of concrete values does not care which line printed first. It only cares whether the bytes existed.
Do not tag the prompt. Tag the argument. Prompts mix policy, examples, and debris. Arguments are the write surface. If the write surface cannot cite a source, the rest of the trace is literature.
A ledger small enough to keep in CI
The artifact below is a proposed runner, not a production SDK. It records tool outputs as a map of field names to values, classifies the next call’s arguments, and emits one JSON span per call. Copy it into a file named assumption_trace.py.
from __future__ import annotations
import hashlib
import json
import time
from dataclasses import dataclass, field
from typing import Any, Callable, Literal
Kind = Literal["observed", "assumed"]
@dataclass
class Source:
tool: str
field: str
digest: str
@dataclass
class Ledger:
values: dict[str, Source] = field(default_factory=dict)
def remember(self, tool: str, payload: Any, prefix: str = "") -> None:
if isinstance(payload, dict):
for key, val in payload.items():
self.remember(tool, val, f"{prefix}{key}." if prefix else f"{key}.")
return
if isinstance(payload, list):
for i, val in enumerate(payload):
self.remember(tool, val, f"{prefix}{i}.")
return
if payload is None or payload == "":
return
text = str(payload)
digest = hashlib.sha256(text.encode()).hexdigest()[:16]
self.values[text] = Source(tool=tool, field=prefix.rstrip("."), digest=digest)
def classify(self, arguments: dict[str, Any]) -> dict[str, dict[str, Any]]:
tagged = {}
for name, raw in arguments.items():
text = "" if raw is None else str(raw)
src = self.values.get(text)
if src is None:
tagged[name] = {"kind": "assumed", "value_preview": text[:80]}
else:
tagged[name] = {
"kind": "observed",
"source_tool": src.tool,
"source_field": src.field,
"digest": src.digest,
}
return tagged
@dataclass
class AssumptionBudget:
max_assumed_fields: int = 0
allow: frozenset[str] = frozenset({"reason", "comment"})
def violations(self, tagged: dict[str, dict[str, Any]]) -> list[str]:
bad = []
assumed = [
name
for name, meta in tagged.items()
if meta["kind"] == "assumed" and name not in self.allow
]
if len(assumed) > self.max_assumed_fields:
bad.append(f"assumed_fields={assumed}")
return bad
class TracedTools:
def __init__(self, impl: dict[str, Callable[..., Any]], budget: AssumptionBudget):
self.impl = impl
self.budget = budget
self.ledger = Ledger()
self.spans: list[dict[str, Any]] = []
def call(self, tool: str, **arguments: Any) -> Any:
tagged = self.ledger.classify(arguments)
started = time.time()
violations = self.budget.violations(tagged)
span = {
"name": f"tool.{tool}",
"ts": started,
"arguments": tagged,
"violations": violations,
"status": "blocked" if violations else "ok",
}
if violations:
span["ended_ts"] = time.time()
self.spans.append(span)
raise PermissionError(f"{tool} blocked: {violations}")
result = self.impl[tool](**arguments)
self.ledger.remember(tool, result)
span["ended_ts"] = time.time()
span["result_type"] = type(result).__name__
self.spans.append(span)
return result
def dump(self, path: str) -> None:
with open(path, "w", encoding="utf-8") as handle:
json.dump({"spans": self.spans}, handle, indent=2)
The important field is not latency. Latency tells you the free endpoint was slow. source_tool tells you whether repo_path was returned by pwd or dreamed up after a truncated thought. When a later diff looks mysterious, open the span first. If kind is assumed, stop reading the prompt. The prompt is downstream of a missing observation.
A fixture that fails on purpose
Wire two fake tools. One lists files. One “patches” a file. The second call should die when the model supplies a path that the listing never produced.
from assumption_trace import AssumptionBudget, TracedTools
def list_files(root: str) -> dict:
return {"root": root, "files": ["app.py", "test_app.py"]}
def patch_file(path: str, comment: str) -> dict:
return {"patched": path, "comment": comment}
def run_good() -> None:
tools = TracedTools(
{"list_files": list_files, "patch_file": patch_file},
AssumptionBudget(max_assumed_fields=0),
)
listing = tools.call("list_files", root=".")
tools.call("patch_file", path=listing["files"][0], comment="keep tests green")
tools.dump("/tmp/trace-good.json")
def run_bad() -> None:
tools = TracedTools(
{"list_files": list_files, "patch_file": patch_file},
AssumptionBudget(max_assumed_fields=0),
)
tools.call("list_files", root=".")
tools.call("patch_file", path="prod.env", comment="looks important")
if __name__ == "__main__":
run_good()
try:
run_bad()
except PermissionError as exc:
print(exc)
Run it as a regression, not as a demo:
python assumption_trace_demo.py
python -c "import json; print(json.load(open('/tmp/trace-good.json'))['spans'][1]['arguments'])"
The good span should show path.kind == observed and source_tool == list_files. The bad run should never reach patch_file’s implementation. That is the whole loop: observe, cite, or stop. If you only log the exception message, you will re-debug the same assumed path next week. Keep the JSON.
Where a free model and a free server fit
The classifier above does not call a model. It should not. Models are good at proposing the next tool. They are unreliable witnesses of their own sources. Use a model to generate candidate calls, then run those calls through TracedTools on your machine. The trace is the contract. The model is the suspect.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access is useful when you want a second proposer for the same ledger: same tools, same budget, different completion. MonkeyCode’s free server option is useful when the assumption audit itself should run off your laptop—span JSON in, violation list out—without promoting the auditor to a production control plane. Neither claim is a quota, a benchmark, or a promise that the remote side will stay up. Treat the remote side as another span source. If it does not return a closed trace, the run is incomplete, not successful.
A practical split looks like this. Keep the ledger on the runner that can see tool bytes. Ship only span summaries if you need a remote reducer: tool name, argument names, kind, source digest, violation strings. Do not ship the raw observation payload unless that payload is already public. Digests compare. Secrets do not need to travel for this check to work.
When the free remote side flakes, you still have /tmp/trace-good.json and a PermissionError. That pair is enough to open a ticket against the agent, not against the network.
What the debug loop looks like on a real failure
Start from the last blocked span, not from the chat. Read violations. If assumed_fields contains sha or url, the model skipped a fetch. Add or force that fetch. Re-run the same fixture. If the field flips to observed and the tool still misbehaves, you have a different bug: a wrong observation, not a missing one. This method does not catch wrong observations. It only catches uncited ones. That limit is the point. Mixed bugs make noisy traces.
Next, diff two traces by argument kind, not by prose. A one-line check is enough:
python - <<'PY'
import json, sys
a = {s["name"]: s["arguments"] for s in json.load(open(sys.argv[1]))["spans"]}
b = {s["name"]: s["arguments"] for s in json.load(open(sys.argv[2]))["spans"]}
for name in sorted(set(a) | set(b)):
left = {k: v.get("kind") for k, v in a.get(name, {}).items()}
right = {k: v.get("kind") for k, v in b.get(name, {}).items()}
if left != right:
print(name, left, "->", right)
PY
/tmp/trace-good.json /tmp/trace-bad.json
If kinds are stable and the user-visible bug remains, stop tuning the tracer. The tracer has done its job. Move to an eval on the observed payloads.
Allow-lists exist because some fields are commentary. comment, reason, and commit_message are often generated on purpose. Put them in allow. Do not put path, owner, endpoint, or sha there because those fields have a source in a healthy run. If your agent must invent a new filename, have it observe the directory first, then propose a name that is not in the listing, and record that proposal as a separate, explicit invented kind. The snippet above does not implement invented. Adding it without a review path just relabels guesswork.
Limits, and who should skip this
Byte equality is brittle. ./app.py and app.py are different strings. Normalize paths and ids before remember, or the ledger will mark real citations as assumed. Normalization belongs in the tool wrapper, not in the model prompt. If you let the model “help” with canonical forms, you are back to uncited arguments.
The budget max_assumed_fields=0 is hostile to drafting agents. Code authors, title generators, and planners invent strings. They should not use this fail-closed setting. They can still emit the tags and review them after the fact. Fail-closed is for agents that mutate repositories, cloud objects, or tickets. If a false block is costlier than a bad write, do not install the raise. Dump the span and page a human.
The method also fails open on empty observations. A tool that returns {} adds nothing to the ledger, so the next call is assumed even if the operator expected a default. That is correct behavior. Empty success is not a source. Fix the tool.
Do not treat span dumps as compliance. Redaction, retention, and access control are separate. This loop only answers whether a write could cite a prior read.
If you already have a parent-span contract, keep it. Causality and provenance are different axes. A child span can be well-parented and still guess its arguments. A guessed argument can sit under a perfect parent and still ship a bad patch. Run both checks. Do not merge them into one attribute just to keep the JSON pretty.
The reusable piece is the ledger, the tags, and the blocked status. Everything else is adapter code. Swap the fake patch_file for your real tool. Keep the raise. When an assumed path shows up, you will spend the next debug session on missing observations instead of on another prompt rewrite. If you want a second proposer against that same ledger, MonkeyCode’s free model access is one place to try it; keep the fail-closed check on your side of the boundary.
Top comments (0)