A while ago I replayed the Claude Code session that built the tool I'm writing about. 906 events. 245 commands that changed something on disk. And 11 failures I never noticed while it was happening.
Nothing crashed. The session looked fine. The agent said it was done, the tests mostly passed, and I moved on. It was only when I stepped through the recording that I saw the commands that had quietly failed and been worked around.
That's the problem I keep coming back to: when an agent does something, the logs usually tell you what the agent reported. Not what actually happened. And they're written by the same system that made the mistake.
So I started building a recorder for agent runs. It captures every LLM call and tool call, links each tool call back to the model turn that asked for it, and lets you replay a run step by step. Along the way I learned that the hardest part isn't recording. It's making sure the recording isn't lying to you.
Here's what I found.
How I checked
I didn't want to just say "supports CrewAI" because the docs say there's a hook. So I built a small harness for each framework:
- a stub model server that returns a scripted tool call, so the run is predictable
- a spy wrapped around the real tool function, counting how many times it actually ran
The spy lives outside my recorder entirely. So at the end of a run I can compare two numbers: how many tool calls really happened, and how many my recorder says happened. If they don't match, the recorder is wrong.
They didn't match. A lot.
1. Tool calls disappearing in threads (CrewAI)
The first time I ran the harness against CrewAI, the spy counted 2 tool calls. My recorder captured 0.
The tools ran. The trace just didn't know about them.
The cause turned out to be a general Python thing. My recorder kept the "current run" in a ContextVar, and context variables don't automatically follow work into a thread pool. CrewAI runs tools in threads, so code inside the tool couldn't see the active trace:
import contextvars
from concurrent.futures import ThreadPoolExecutor
current_run = contextvars.ContextVar("current_run", default=None)
def tool():
return current_run.get()
current_run.set("run_123")
with ThreadPoolExecutor() as pool:
print(pool.submit(tool).result()) # None: context is lost
ctx = contextvars.copy_context()
print(pool.submit(ctx.run, tool).result()) # run_123: context kept
After fixing propagation into threads, a single tool call worked. Then a thread-plus-pool scenario came back as 1 of 4 until I switched to a ContextVar stack.
If your tracing shows the LLM calls but the tool calls are missing, this is the first thing I'd check.
2. Recording <APIResponse [200 OK]> instead of the response
This one was embarrassing. Under LangChain and CrewAI, some calls go through the SDK's raw-response wrapper. My recorder was happily capturing the wrapper object, so the "response" field in my traces said:
<APIResponse [200 OK]>
No content. No error either. Every trace looked complete. You'd only notice if you actually read a response.
That's the kind of bug that makes me nervous about any tracing tool, mine included. A missing field is obvious. A field that's filled with the wrong thing isn't.
3. Token counts ~40x too high
My token accounting summed cache_read_input_tokens across every turn of a conversation. With prompt caching, each turn re-reads a large cached prefix, so adding those up turn after turn counted the same context over and over. The usage it reported was roughly 40x too high.
If you build cost tracking on top of agent traces, double-check what you're summing. This is an easy mistake to make, and a confusing one to debug, because the numbers look plausible on any single turn.
4. The hook that never fires (Pydantic AI)
Pydantic AI has a method called handle_call that looks exactly like where tool calls go through. I hooked it. Nothing.
So I spied on all four candidate methods and checked which ones actually ran. The real path is execute_tool_call. handle_call never fired in my testing.
The lesson for me: don't trust a method name. Instrument, then verify against something you didn't write.
5. Bonus: why json.dumps broke my hash chains
Every event in a trace is hashed, and each hash covers the previous one, so if anyone edits, removes or reorders an event later, verification fails at that point.
It worked perfectly in Python, and failed in every other language. The reason:
>>> json.dumps(50.0)
'50.0'
> JSON.stringify(50.0)
'50'
Same value, different bytes, different hash. A trace written in Python couldn't be verified anywhere else. The fix was RFC 8785 canonical JSON, and I now check every trace against a Node implementation in CI.
What I took away
Every one of these bugs had the same shape: the trace looked fine. No errors, no gaps you'd spot at a glance. Just quietly wrong.
And that's a bit uncomfortable, because the whole reason you record an agent run is so you can trust it later, when something has gone wrong and you need to know what really happened.
The tool I ended up with is called Deposition. It records LLM calls and tool calls into a hash-chained trace, with optional signing, links tool calls to the model turns that requested them, and replays runs locally. It also has a way to declare what a run was allowed to do (tools, cost, steps) and audit the run against it afterwards, and it can import Claude Code sessions, which is how I found those 11 failures.
Some honest limits:
- Streaming responses record the request but not the content yet.
- It's tamper-evident, not tamper-proof. A signing key on the same machine raises the bar but doesn't settle it.
- Audits happen after the run. It doesn't stop an agent at runtime.
- Framework hooks move. I re-verify against current releases rather than assuming they still work.
It's open source (Apache 2.0), with zero required dependencies: github.com/chiragrajputuss-code/deposition
I'd genuinely like to hear from people running agents in production: when something goes wrong, what do you need to prove about the run, and to whom? That question is what I'm trying to build toward, and I suspect the answers are more varied than I think.
Top comments (4)
For the 40× figure, are you comparing unique context size or usage summed across requests? A two-turn example with cache-read, cache-write and uncached input tokens shown separately would help clarify what the corrected counter measures.
Fair question. It's usage summed across requests, both before and after, so neither number is unique context size.
Here are two real turns. Turn 1 had 2 uncached input tokens, wrote 14,754 to the cache, read 22,858 from it, and output 534. Turn 2 had 2 uncached, 1,070 written, 37,612 read and 182 output. Turn 2 is basically re-reading everything turn 1 already saw.
The old counter added all of it on every request, which comes to about 77k. The fixed one skips cache reads and comes to about 16.5k. So that's 4.7x after two turns, and the gap keeps growing because every turn re-reads the whole context. On a long session it ended up around 40x.
Neither number is cost, since cache reads and writes are priced differently. The trace keeps all four raw numbers so you can work that out yourself.
Curious, how are you tracking token usage on your own agents? Summing usage fields, or something else?
The hard part with agents is not making them act, it is making them stop and ask.
Agreed, and Deposition doesn't do that part. It records after the fact and never blocks anything. Where it helps is showing where a run should have stopped. In the bundled demo, the agent reads a fetched page and then calls send_email with an address that came from that page. That's exactly the "should have asked" moment, and it's flagged because the run's mandate declared send_email external. A runtime gate needs the same list. Deposition just checks it afterwards instead of enforcing it live.
Out of interest, have you found a good way to decide when an agent should stop and ask? Hardcoded rules, model judgment, something else?