Agent Runs Are Black Boxes: A Trace-First Debug Loop
As agents take on more coding work, the developer's job shifts from writing to reviewing. Reviewing without a trace is guessing. When an agent misbehaves, the instinct is to edit the prompt — but the prompt is the wrong artifact. The run is the artifact: the ordered list of tool calls, diffs, and token spends that actually happened. Trace it before you tune it.
An agent is not a function. A function maps an input to an output; an agent maps a task to a sequence of side effects, and each side effect changes the next decision. Rerun the same prompt and you may get a different failure. That is why "it worked in my terminal" is not a debugging strategy, and why screenshots do not help: the interesting state is spread across ten steps, not one answer.
Debugging an agent without a trace is like debugging a network outage with ping alone. You know the packet left. You do not know where it died.
Here is a minimal, reusable trace harness. It writes one JSON event per line to an append-only JSONL file. Three event types matter: model calls, tool calls, and tool results — plus a diff snapshot after every tool execution. A preflight harness tells you whether the agent can start; this trace tells you what it did after. This is a minimal harness, not a library; adapt attribute names to your SDK.
# trace_agent.py — minimal run tracer for tool-using agents
import json, time, uuid
from pathlib import Path
def emit(entry, trace_path):
with open(trace_path, "a") as f:
f.write(json.dumps(entry) + "\n")
def trace_run(agent, task, trace_dir="traces"):
run_id = uuid.uuid4().hex[:8]
trace_path = Path(trace_dir) / f"{run_id}.jsonl"
trace_path.parent.mkdir(parents=True, exist_ok=True)
emit({"event": "run_start", "run_id": run_id, "task": task}, trace_path)
messages = [{"role": "user", "content": task}]
for step in range(10):
t0 = time.time()
response = agent.complete(messages) # any OpenAI-compatible client
emit({
"event": "model_call", "step": step, "run_id": run_id,
"latency_s": round(time.time() - t0, 3),
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"content": response.content,
}, trace_path)
if not getattr(response, "tool_calls", None):
break
for call in response.tool_calls:
emit({"event": "tool_call", "step": step, "run_id": run_id,
"tool": call.function.name,
"args": call.function.arguments}, trace_path)
result = execute_tool(call) # your dispatcher
emit({"event": "tool_result", "step": step, "run_id": run_id,
"tool": call.function.name, "ok": result.ok,
"preview": result.preview[:500]}, trace_path)
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": result.preview,
})
emit({"event": "run_end", "run_id": run_id, "steps": step + 1}, trace_path)
The harness is provider-agnostic. It accepts any OpenAI-compatible endpoint, including MonkeyCode's free model access — the environment this article is about. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project; at the time of writing, its free tier includes 10 million tokens and a free server option for running the agent loop itself. No latency or reliability benchmarks are included here. Quotas change, so verify the current terms before building on them.
Once you have a trace, read it with three questions. First: did the tool call match the intent? Second: did the result get used, or ignored? Third: did the diff move toward the goal? Two shell one-liners and a git command answer all three.
jq -r 'select(.event=="tool_call") | "step \(.step): \(.tool) \(.args)"' traces/*.jsonl
jq -r 'select(.event=="tool_result") | "step \(.step): ok=\(.ok) preview=\(.preview)"' traces/*.jsonl
git diff --stat
The diff snapshot is the part most people skip. Snapshot git diff --stat before each tool call and after it. If the diff grows while the task says "refactor," you are watching churn, not progress. If the diff never changes while tokens climb, the agent is in a read-only loop. A repeated tool call with identical arguments shows up in the trace as an exact duplicate event — no log aggregation tool required.
A common failure pattern this loop exposes looks like this. The agent is asked to find a failing test and fix it. The trace shows a search tool called with the same query three times, an empty result each time, and then a model call that invents a file path. The fix is not a better prompt. The fix is a guard: break the loop when a tool result exactly matches the previous one. The trace turns a "sometimes it fails" mystery into a two-line patch.
Store traces per task, not per day. When a failure reappears, diff the two JSONL files; the first divergent event is your bug. That comparison is the debug loop, and it takes seconds. The trace is also a token ledger: when a task that should cost two thousand tokens costs forty thousand, the trace shows where the spend went — usually a loop, not a single expensive call.
That is the payoff. Failures become reproducible categories instead of moods. You stop asking "why did the model do that?" and start asking "which step diverged?" The second question has an answer you can act on.
Limitations are real. A trace records behavior, not reasoning. You will see the tool call, not the chain of thought behind it; if you need that, use a provider that exposes reasoning content. A JSONL file is not a metrics platform; for a team, export the same events to Langfuse or OpenTelemetry. And one trace is a sample, not a proof — run the same task three to five times before concluding, because agents are non-deterministic by design.
Who should not use this? If your agent is a single model call with no tools, a trace is overhead; print the response and move on. If you already run a full observability stack, use it instead of a local file. And if you are building production traffic on a free tier, the trace will tell you when you outgrow it — that is exactly what it is for.
The harness above is the whole method. If you want to run it against a free endpoint, MonkeyCode's free model access and free server option are a low-cost place to start. The trace will tell you when you need something better.
Top comments (2)
The piece I would add is a tiny negative trace. Recording what the agent considered and did not do catches a different class of bug than tool-call logs alone. The weird failures in my own agent runs usually come from a skipped boundary check, not from the final patch being hard to read.
The strongest part here is treating the run as the unit of debugging, but I’d push it one step further: don’t just trace tool calls trace decision boundaries. A repeated tool call isn’t necessarily the failure; the important question is why the agent remained eligible to make that call after the result provided no new information. That suggests tracking state-change-per-step, not just token spend and diffs. Once you measure “new information gained” against “side effects introduced,” you can detect loops, churn, and premature tool escalation much earlier. That’s the kind of observability pattern that becomes especially valuable when hardening agentic workflows at IT Path Solutions.