Traced a Multi Agent Failure Back 6 Steps. Here's the Instrumentation That Made It Possible.

Last week an agent in a production pipeline I was debugging did something that looked, on the surface, completely irrational. Correct tool call, correct parameters, valid output, but the decision to make that call at all was wrong. It took six steps of tracing backward to find out why.
Here's the actual root cause, and why most observability setups wouldn't have caught it.
The setup: a multi agent pipeline where Agent A researches a topic, writes findings to shared memory, and Agent D (four hops later, unrelated task) reads from that same memory scope because the key happened to overlap. Agent A's findings were accurate when written. By the time Agent D read them, the underlying data had changed. Agent D reasoned perfectly, off information that was stale by the time it mattered.
Why this is hard to catch:
The failing agent's logs look completely normal, valid input, valid reasoning, valid output
Standard tracing shows what happened at each step, not when a piece of context was written versus when it was consumed
Nobody flagged it because no individual step was wrong, the failure only exists in the relationship between two steps that happened at different times
What actually solved it:
Timestamp every memory write and read separately, and diff the gap when a failure investigation starts, a large gap between write and read is often the first real clue
Scope memory access explicitly rather than relying on implicit key matching, Agent D shouldn't have been able to see Agent A's write in the first place if the scopes were correctly separated
Make replay possible from any single node, so you can re-run just the suspect step with the memory state as it existed at read time, not the current state
This kind of cross temporal bug is becoming more common as pipelines get longer and memory gets shared across more agents. Most tracing tools are built for single agent debugging and don't surface this class of issue well.
We ended up building this kind of instrumentation into Cartha, our own agent governance layer, after running into this exact problem across a few different customer pipelines. Curious if others have hit similar cross temporal failures, or if this is more of an edge case in typical setups.