I spent two hours staring at a terminal that said nothing useful.
The agent had "succeeded." Status: green. Output: a half-written PR description, a deleted test file it was supposed to fix, and a cheerful summary claiming everything was fine. No stack trace. No failed step. No breadcrumbs. Just vibes.
If this were an API, I'd open the logs, grep the request id, and find the 500 in twelve seconds. With agents, I get a shrug and a bill.
That's the observability gap — and it's why most "autonomous" demos die the moment you try to ship them.
Agents fail like interns, not like services
A microservice fails loudly. Status codes. Timeouts. Structured errors. You can alert on them.
An agent fails politely. It retries the wrong tool. It invents a path that doesn't exist. It loops on a "almost right" answer until your token budget is a sad little crater. Then it writes a confident final message like a student who didn't do the reading.
Think of a normal backend as a subway map: every station reports where the train is. An agent without traces is a paper airplane you threw over a fence. You know it left. You do not know if it hit the neighbor's dog.
Stop debugging with "print(thoughts)"
The first instinct is to dump the chain-of-thought into stdout and squint. That works once. It does not work at 2 a.m. when three tools fired, two of them twice, and the model quietly changed the plan mid-flight.
You need the same three things every debuggable system has:
- Identity — a run id for the whole job
- Steps — ordered spans: plan → tool call → tool result → decision
- Outcomes — success / retry / give-up, with why
Not a novel. A flight recorder.
A pasteable mini flight recorder
Drop this next to your agent loop. It is deliberately boring. Boring is the point.
import json, time, uuid
from contextlib import contextmanager
class AgentTrace:
def __init__(self, job: str):
self.run_id = str(uuid.uuid4())[:8]
self.job = job
self.events = []
self.t0 = time.time()
def log(self, kind: str, **data):
self.events.append({
"ts": round(time.time() - self.t0, 3),
"run": self.run_id,
"kind": kind,
**data,
})
@contextmanager
def span(self, name: str, **meta):
self.log("span_start", name=name, **meta)
t = time.time()
try:
yield
self.log("span_ok", name=name, ms=int((time.time()-t)*1000))
except Exception as e:
self.log("span_err", name=name, error=str(e)[:300])
raise
def dump(self, path=None):
payload = {"job": self.job, "run_id": self.run_id, "events": self.events}
text = json.dumps(payload, indent=2)
if path:
open(path, "w").write(text)
return text
# usage inside your loop
trace = AgentTrace("fix-flaky-test")
with trace.span("plan"):
plan = llm.plan(ticket) # your planner
trace.log("plan", steps=plan)
for step in plan:
with trace.span("tool", tool=step["tool"], input=step.get("input")):
result = tools[step["tool"]](step.get("input"))
trace.log("tool_result", tool=step["tool"],
ok=bool(result), preview=str(result)[:200])
if not result:
trace.log("decision", action="retry_or_abort", reason="empty_result")
break
print(trace.dump("last_run.json"))
Now when the agent "succeeds" while deleting your tests, you open last_run.json and see the exact span where the plan went feral — instead of re-running the whole thing and hoping the model feels different today.
Three signals worth alerting on
You do not need a full OpenTelemetry cathedral on day one. You need three cheap tripwires:
- No-progress loop — same tool + same args ≥ N times → abort
- Silent success — final message says done, but zero write-tools fired → flag it
- Budget burn — tokens or wall-clock past a cap mid-job → checkpoint and stop
These are the agent equivalents of 5xx rate, p99 latency, and disk-full. If you only log "final answer," you are flying without instruments.
Why this beats "just add more prompts"
Prompting harder is like yelling directions at a delivery driver who has no GPS trail. Sometimes they arrive. You still cannot tell which turn went wrong when they do not.
Traces turn "the agent is weird" into "step 4 called rm with a bad path because the planner hallucinated a folder." That is a fixable bug. Vibes are not.
Same lesson as last month's tools-over-prompts rant, different layer: surface area you can inspect beats eloquence you cannot.
A 30-minute upgrade path
- Wrap every tool call in a span (name, args hash, ok/error, ms).
- Persist one JSON artifact per run — even a local file is fine.
- Add the three tripwires above as hard stops, not log lines.
- When a run looks cursed, paste the trace into your next prompt as evidence, not as a novel.
You just turned a black box into a system you can shame in code review.
The punchline
APIs earned our trust because they fail in public. Agents will earn it the same way — not with longer system prompts, but with boring, greppable, slightly embarrassing logs.
If your agent cannot explain what it did in a JSON file a human can skim in under a minute, it is not autonomous. It is unsupervised.
Your turn: open the last agent run you are half-proud of. What is the one field you wish had been in the log — tool args, token burn per step, plan diffs, something else? Drop the field name in the comments. I am collecting a minimal "agent flight recorder" schema and I will steal the good ones.
Top comments (2)
The framing that landed for me is that an agent fails politely while a service fails loudly — nothing raises, nothing 500s, it just produces a confident summary of work it never did. Your "silent success" tripwire (final message says done, zero write-tools fired) is the cheapest possible defence against that and it's remarkable how often it's missing: most harnesses check that the loop terminated, not that anything happened.
I'd push a little on the no-progress detector though. Same-tool-same-args catches the obvious loop, but the interesting failure is an agent that varies its arguments just enough to look like progress while re-deriving the same conclusion each time. Have you tried keying the detector on the result rather than the call — e.g. same tool plus an unchanged diff/output hash — or does that generate too many false positives on legitimate retries?
"An agent fails politely" is painfully accurate. The failure mode that cost me the most time was the confident final message after a silently abandoned subtask - the trace showed success because the agent reported success. What finally helped: tracing tool calls as first-class events and alerting on the gap between steps planned and steps actually executed, not on errors. Agents almost never throw; they just quietly do less than they claimed.