Observability for AI agents without instrumentation: read the transcript

Most observability advice starts the same way: wrap your calls, add a tracer, emit spans, ship them somewhere. That is a lot of work to do before you have learned anything, and it only helps the runs you thought to instrument ahead of time. The run you actually want to understand is usually the one that already happened, the one you did not plan for.

Here is the thing I kept forgetting: the transcript is already the source of truth. Claude Code writes every session to disk as it goes, at ~/.claude/projects/<slug>/<session-id>.jsonl. Every subagent delegation, every prompt, every result, every tool call. It is all there whether or not you set anything up in advance. So the whole "add instrumentation" step is skippable. You do not need to trace the run. You need to read the file the run already left behind.

I built two small tools on that idea. They read local files, no API keys, no network, no SDK. One asks "which of these results should I not trust?" and the other asks "what filled up the context window and what can I delete?"

Tool one: flagging runs you should not trust

The first tool is agentrace. Directing agents turned out to be the easy half of the job. The hard half is verification. When you fan out ten background subagents and each returns a confident wall of text, generation is not the bottleneck. Knowing which answers count as proof is. And you cannot verify what nobody reads. By the time the agents report back, the interesting parts are buried in a transcript nobody opens.

So agentrace reads it for you and flags the suspicious results. Every check exists because it actually bit me, over roughly 150 research subagents across two weeks. A few of the ones that earned their place:

• error: agents dying on session limits mid-sweep, work silently lost, nobody noticing until the final report came back short.
• absence_as_evidence: an agent concluded a company was not hiring because an API returned an empty list. That API returns empty with HTTP 200 for accounts that do not even exist. Absence of data is not evidence of absence.
• gave_up: "I was unable to find..." reads like an answer if you skim. It is not one.
• hedged_claim: an agent said a company "appears to be" hiring. That hedge quietly became a fact by the time it reached a decision. The hedge was honest; the bug was flattening it downstream.
• unverified_urls: twenty URLs cited, none opened. That is autocomplete, not research.
• no_output_contract and thin_prompt: the failure that is mine, not the model's. A task with no definition of done cannot be verified, because I never really asked the question.

Run against the exact session that motivated the tool, 152 subagent runs, it flagged 36 of them, with 39 findings total: 7 agents that died on session limits, 17 hedged claims, and 12 prompts where I forgot to specify an output shape.

That last number is the one I keep coming back to. Twelve of the flagged runs were not model failures at all. They were me handing an agent a task with no shape and then being surprised the result had no shape. Most agent tooling assumes the model is the problem. A lot of the time the prompt is.

Tool two: profiling the context window

The second tool is ctxlens, a flamegraph for the context window. Agents get slow, expensive, and frankly dumb when their context fills with junk: the same file read six times, a 12k-token tool result that mattered for exactly one turn, tool schemas re-sent on every single step. Token dashboards tell you the bill. They do not tell you where the bytes went or what to cut.

ctxlens parses the same transcripts and gives a per-turn breakdown. Every message is attributed to a segment: system, tool_definitions, user, assistant, thinking, tool_call, tool_result. Then it computes a waste ratio, total_waste / total_tokens, summing duplicate tokens, tool-result bloat above a cap, stale outputs, and oversized tool definitions. On one session it profiled 12,481 tokens across 14 turns and called out 4,932 of them, 39.5 percent, as waste.

The recommendations are rule-based and specific, not generic advice:

• Repeated content: the same file or tool result appearing more than once. One session had Read:file_path=config.py show up six times across turns 2, 5, 7, 9, 11, and 13. Every copy after the first is dead weight.
• Stale tool outputs: an older result superseded by a newer one for the same target, still occupying context.
• Oversized tool definitions: schemas above budget, paid on every turn.
• Tool results dominate and single biggest consumer: callouts when one category or one message is eating the window.

Each recommendation carries a severity and an estimated token saving, so you can act on the expensive ones first. It works offline with a deterministic heuristic tokenizer for relative profiling and CI, and upgrades to exact BPE counts automatically when tiktoken is installed. That fallback matters more than it sounds: it means the tool runs anywhere with no heavy dependency, and you only pay for tiktoken when you need the exact number.

The one honest caveat

These are heuristics over text. That is the whole trick and also the whole limitation. agentrace tells you what to go read; it does not tell you what is true. A checker that cries wolf gets switched off, which is worse than no checker at all, so severity is deliberately conservative. thin_prompt used to fire on any prompt under 200 characters, until I noticed it was flagging perfectly verifiable short prompts and teaching nobody anything. Now a prompt has to be short and never say what done looks like before it fires. On the bundled fixture that dropped findings from 16 to 9 without losing a single true one.

Same honesty applies to ctxlens. A heuristic tokenizer is great for relative comparisons and catching duplication; it is not the exact bill. Treat both tools as a fast way to point your attention, not as a verdict.

What this buys you

No instrumentation. No wrapper. No SDK. The data is already on disk, so you can analyze the run you wish you had traced, after the fact, on any session, including the one still being written (both tools skip a torn final JSON line instead of refusing to parse). That is the part I did not expect going in: the observability was never missing. It was sitting in a .jsonl file the whole time, waiting for something to read it.

Both tools are MIT licensed and up at github.com/royalpinto007. If you run agents at any scale, point them at your ~/.claude/projects directory and see what falls out.