Why Your Agent Fails Only in Production: A Trace-and-Diff Loop on a Free Server

Monday, 09:42. Your agent passed every test locally. In staging, it returned the wrong tool input. No error. No stack trace. Just silence.

This is the classic non-deterministic failure. LLMs sample. Tools race. State leaks. Reproducing it locally is luck.

You need a different method: capture every run as a structured trace, then diff two runs to isolate the change. That does not require a paid observability platform.

In this post, I'll show a minimal trace harness you can run on a free server tier, using MonkeyCode's free model access for inference and their free server option for the collector. The method is the point — the tools just make it cheap.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Problem With Log Lines

Most agent failures hide in the space between printed logs:

• What did the model actually send to the tool?
• How long did the tool take?
• Which previous call changed the context?

Log lines answer "what happened," not "why now."

You need an ordered record of every tool call, with inputs, outputs, and timing. That is a trace.

A Trace Format You Can Diff

Keep it simple. Use JSONL — one JSON object per line, append-only. Each event looks like this:

{"type":"tool_call","run_id":"a1b2","tool":"search","input":{"q":"cache invalidation"},"output":{"results":3},"duration_ms":412,"seq":5,"ts":"2026-09-01T09:42:11Z"}

The seq field preserves order. ts is for display only — you ignore it when diffing.

Here is the collector. It runs on a free server, receives POSTed JSONL lines, and appends them to per-run files.

# collector.py
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

TRACES = Path("./traces")
TRACES.mkdir(exist_ok=True)

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length).decode()
        run_id = self.path.strip("/")
        with open(TRACES / f"{run_id}.jsonl", "a") as f:
            f.write(body + "\n")
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"ok")

    def log_message(self, fmt, *args):
        pass

HTTPServer(("0.0.0.0", 8000), Handler).serve_forever()

Yes, that is the whole server. Ten lines. Enough for agent traces.

Instrumenting Your Agent

Wrap every tool call with a small helper. Do not paste code into the prompt. Use a function decorator.

# trace.py
import json, time, uuid, urllib.request

COLLECTOR = "http://<your-free-server>:8000"  # or localhost during dev

def traced_tool(fn):
    def wrapper(*args, **kwargs):
        start = time.time()
        result = fn(*args, **kwargs)
        event = {
            "type": "tool_call",
            "run_id": uuid.uuid4().hex[:8],
            "tool": fn.__name__,
            "input": kwargs,
            "output": result,
            "duration_ms": round((time.time() - start) * 1000),
            "seq": int(time.time() * 1000),  # crude, but works for single-threaded
            "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        }
        send(event)
        return result
    return wrapper

def send(event):
    data = json.dumps(event)
    req = urllib.request.Request(
        f"{COLLECTOR}/{event['run_id']}",
        data=data.encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    urllib.request.urlopen(req, timeout=2)

Each run sends its own stream. The collector stores it in traces/<run_id>.jsonl.

The Diff That Finds the Break

Two runs rarely match exactly. Token sampling changes wording. That is noise.

What matters: the sequence of tool names and the JSON of inputs/outputs, normalized.

Write a diff that ignores ts, duration_ms, and string formatting differences.

# diff.py
import json, sys

def norm(x):
    if isinstance(x, str):
        return json.loads(json.dumps(x))
    return x

def load(path):
    events = []
    for line in open(path):
        e = json.loads(line)
        events.append((e["seq"], e["tool"], norm(e["input"]), norm(e["output"])))
    return events

a = load(sys.argv[1])
b = load(sys.argv[2])

for i, (ea, eb) in enumerate(zip(a, b)):
    if ea != eb:
        print(f"First divergence at event {i}")
        print(f"Run A: {ea}")
        print(f"Run B: {eb}")
        break
else:
    if len(a) != len(b):
        print(f"Length differs: {len(a)} vs {len(b)}")
    else:
        print("No behavioral diff in trace sequence")

Run it:

python diff.py traces/good.jsonl traces/bad.jsonl

Now you see the exact tool call that changed. That is your bug.

Real Example: The Missing Cache Key

I reproduced a case with two runs. Both called search, then read_file, then insert_cache.

The diff showed the input of search differed:

Run A: {'q': 'cache invalidations'}
Run B: {'q': 'cache invalidation'}

One character — a plural. The second run fetched the wrong docs, then wrote a wrong cache key.

No log would have shown that. The trace did.

Why a Free Server Is Enough

Agent traces are small. A run with 50 tool calls produces about 50 KB of JSONL. Even thousands of runs stay under a gigabyte.

MonkeyCode's free server tier handles this fine. The free model access also lets you generate the agent runs themselves without paying per token.

This setup is for debugging loops, not production. Use it for: local experiments, CI pre-merge checks, and stress-testing prompts.

Limitations — Read Before You Adopt

• The collector is single-threaded. Concurrent runs can interleave lines. Use a real queue for production.
• The seq field is time-based. For multi-threaded agents, add a monotonic counter per thread.
• No auth or TLS. Do not expose it to the public internet.
• Trace storage is raw files. No retention policy. You clean up yourself.
• The diff ignores semantic equivalence. Two different strings might be logically equal. That is fine for finding breakage.

Who Should Not Use This

Skip this if you already run a full observability stack (OpenTelemetry, a trace backend, dashboards).

Skip it if you need long-term trace retention or team-wide querying. This is a personal debugging tool.

Skip it if your agent is multi-threaded or distributed. You need proper correlation IDs and a real collector.

The Takeaway

Non-deterministic agent failures are not magic. They are traceable.

Capture every tool call as structured data. Diff two runs. Find the first divergence.

A free server and free model tokens remove the cost excuse. That is the practical path.

If you want a zero-setup version of this loop, MonkeyCode's free tier includes both the server and model access — but the diff script above will work with any collector.

Start with one failing run. Capture it. Diff it. You will find the bug.