Trace First, Blame Later: A Debug Loop for Failing Agent Runs

An agent run returned exit code 0. The task was simple: update the rate limiter.

The diff showed 400 deleted lines. It also showed a new file named done.txt. No error. No warning. No trace.

The model was not the problem. The observability was.

Every week, teams debate whether the model or the harness failed. This loop settles the debate. Trace first, blame later.

This article is a workflow, not a tool review. Use it with any agent framework. You need a git repo and a way to see tool calls.

Why Logs Are Not Enough

Logs tell you what the agent printed. They do not tell you what the agent did.

Tool calls are the real story. Each call has a name, an input, an output, and a duration. Diffs show the side effects. Together, they form a trace.

You need three signals:

1. Logs — what the agent said.
2. Tool calls — what the agent did.
3. Diffs — what the agent changed.

Capture all three. Correlate them by timestamp. That is the minimum viable trace.

Reviewers need evidence, not vibes. A trace bundle is that evidence.

Prerequisites

You need three things before starting.

1. A git repository. The diff signals need version control.
2. A tool-call stream. The agent must expose its actions.
3. A repeatable command. The same input must start the run.

Without these, the loop collapses. Add them before you debug.

The Artifact: A Trace Bundle Script

The script below wraps any agent command. It produces a timestamped bundle in traces/.

#!/usr/bin/env bash
# trace_bundle.sh — capture logs, tool calls, and diffs from one agent run
# Usage: ./trace_bundle.sh <command...>
set -uo pipefail

RUN_ID="run-$(date +%Y%m%d-%H%M%S)"
OUT_DIR="traces/$RUN_ID"
mkdir -p "$OUT_DIR"

# 1. Snapshot the working tree before the run
git diff > "$OUT_DIR/before.patch" 2>/dev/null || true

# 2. Run the command and timestamp every line
#    Requires `ts` from moreutils. Swap for a while-read loop if needed.
"$@" 2>&1 | ts > "$OUT_DIR/agent.log"
RUN_EXIT=${PIPESTATUS[0]}

# 3. Snapshot the working tree after the run
git diff > "$OUT_DIR/after.patch" 2>/dev/null || true

# 4. Copy the tool-call stream if the agent writes JSONL
#    Point TRACE_FILE at that file before running.
if [[ -n "${TRACE_FILE:-}" && -f "$TRACE_FILE" ]]; then
  cp "$TRACE_FILE" "$OUT_DIR/tool_calls.jsonl"
fi

echo "$RUN_EXIT" > "$OUT_DIR/exit_code"
echo "bundle: $OUT_DIR exit=$RUN_EXIT"

The bundle gives you four files. before.patch and after.patch show side effects. agent.log shows the narrative. tool_calls.jsonl shows the actions.

Most agent frameworks can write a JSONL tool stream. Set TRACE_FILE to that path. If your framework cannot, wrap the tool dispatcher.

Summarize the Tool Stream

A raw JSONL file is hard to read. This small script summarizes it.

#!/usr/bin/env python3
"""Summarize a tool-call JSONL stream."""
import json
import sys
from collections import Counter

calls = []
for line in open(sys.argv[1]):
    try:
        calls.append(json.loads(line))
    except json.JSONDecodeError:
        continue

print(f"tool calls: {len(calls)}")
print("by tool:", dict(Counter(c.get('tool', '?') for c in calls)))
print("by status:", dict(Counter(c.get('status', '?') for c in calls)))

Run it after every failed run. The summary shows the shape of the failure. Then open the first suspicious call.

Reading a Bundle in Practice

Here is a failure pattern that shows up in bundles. The log looks clean. The exit code is 0.

The tool stream tells the real story. The agent called read_file on the same path six times. Each call returned the same content. Then it called write_file with a truncated version.

The diff confirms it. 300 lines vanished. The log never mentioned the truncation.

Correlate by timestamp. The log line at 14:02:11 should match the tool call at 14:02:11. When they disagree, trust the tool call.

This pattern has a name: silent truncation. The model ran low on context but kept going. The framework saw a completed run, so it logged no error.

How do you find the first wrong call? Start from the end. Walk backward until the diff stops matching the tool stream. The mismatch is the failure point.

The Debug Loop

Four steps. Repeat until the bundle looks clean.

1. Reproduce. Run again with the bundle script. Same input. Same environment.
2. Reduce. Cut the task to the smallest failing slice. Remove tools. Remove steps.
3. Isolate. Find the first wrong tool call. Compare its input and output. Check the diff at that timestamp.
4. Fix. Change one variable: prompt, tool, or model. Re-run. Keep the bundle.

Each iteration costs tokens. A failed run is not free. That cost shapes how often you debug.

Token Cost of the Loop

Each loop iteration has a measurable cost. Count tokens per run, not per conversation.

A simple way: read the token usage from the final response metadata. Most frameworks expose it.

run-1: 12,400 tokens — reproduce
run-2: 8,100 tokens — reduce
run-3: 6,300 tokens — isolate
run-4: 5,900 tokens — fix attempt
total: 32,700 tokens

The pattern matters more than the total. A shrinking number means the loop is working. A flat number means you are guessing.

Track this number across runs. It turns debugging into a measurable process.

Keep the numbers in a CSV if you want a trend line. One column per run. One row per loop.

Where Free Resources Fit

MonkeyCode is an open source project with two useful defaults for this loop: free model access and a free server option. The project currently offers a 10M token allowance and a free server for experiments. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server is a low-risk place to run the reproduce step. You can break it without touching production. The token allowance covers the repeated re-runs the loop requires.

Check the project README before relying on the numbers. Quotas and terms change.

A Decision Table for Symptoms

Use this table when a run fails. It maps symptoms to the first thing to check.

Symptom	Likely cause	Check first
Exit 0, wrong diff	Tool misuse	Tool call before the bad diff
Repeating the same call	Prompt ambiguity	Identical tool calls in the stream
Long silence, then timeout	Token budget	Token usage per step
Refusals after a big output	Context overflow	Size of the last tool output

The table is a starting point, not a verdict. The trace decides.

Add your own rows as you collect bundles. After ten runs, you will have a local map of failure modes. That map is more useful than any generic checklist.

Limitations

Three limits. Know them before you run.

First, the loop needs reproducibility. Flaky network calls break it. Pin your tools and inputs.

Second, diffs miss in-memory state. A wrong decision may leave no file change. Log decisions too.

Third, the free tier is for experiments. Do not run production batch jobs on it.

Who Should Not Use This

This loop is for debugging, not production monitoring. If you need alerting, SLAs, and retention, use a real observability stack.

If your agent framework lacks tool-call hooks, the script will not help. Check the framework's tracing API first.

The Takeaway

Next time an agent fails, do not blame the model. Do not blame the harness.

Trace first. Then decide.

The bundle script is small. The loop is repeatable. On a free token allowance, the cost of re-running is close to zero.

If you want to try the loop, MonkeyCode's free model access and free server are a reasonable starting point. Read the README for current limits.