An AI agent audit trail has to answer more than "what did the agent do?", it has to prove "why did it decide that, and what did a bad data source touch?". This post records the real reasoning chain automatically (zero changes to your tools), stores it in Neo4j with the graph vendor's own agent-memory SDK, and runs the reverse audit: when a source turns out wrong, one graph traversal returns every decision that touched it, at read time, where a flat log would scan every record.
Clone and star stop-ai-agents-losing-memory-sample-for-aws
Ask your agent "why did you recommend that flight?" a week later and it will give you a confident, plausible answer. The problem: it's made up.
The real reasoning chain (which tools ran, what sources they read, what the decision rested on) was never kept. The model confabulates a justification because that's what models do when the trace is gone.
Your logs won't save you either. Logs record that things happened. An audit trail for an AI agent has to answer two harder questions:
- The replay: "Why did you decide X?" The real chain, not a reconstruction.
- The reverse audit: "This data source turned out to be wrong. Which of my decisions touched it?"
This post builds both from a live agent session (nothing is scripted, the recorder captures whatever the agent actually did): decision traces captured automatically with zero changes to your tools, and a reasoning graph, stored with Neo4j's official agent-memory SDK, where the reverse audit is a single traversal. The code uses Strands Agents; the pattern carries over to any agent framework.
(Part of the agent-memory series. The intro maps all the memory types. Earlier posts store what the agent knows*; this one stores why it* decided*.)*
What is a "flat" memory? A store that keeps each record on its own, with no edges to traverse between them: a key-value store, a log file, a vector store. As Neo4j puts it, a flat log records what happened; a graph records why. The contrast in this post is a flat trace store (
agent.state) versus a graph (Neo4j).
Why Strands Agents for this demo?
Strands provides the mechanism that makes this possible: a hook system. You register a HookProvider on the agent and it receives the agent's own lifecycle events, BeforeInvocationEvent, AfterToolCallEvent, AfterInvocationEvent, as the agent runs. Crucially, those events carry the data you need: AfterToolCallEvent exposes event.tool_use (the tool name and its input). That is Strands doing the wiring.
DecisionTraceRecorder is the small library I built on top of that mechanism. It is not part of Strands. It is a HookProvider that subscribes to those three events and turns them into a decision trace: open a trace on invocation start, append one step per tool call (reading event.tool_use), close it with the outcome on invocation end.
from strands import Agent
from trace_kv import DecisionTraceRecorder # the recorder I built with Strands hooks
agent = Agent(
model=model,
tools=[search_flights, check_fare_alert],
hooks=[DecisionTraceRecorder()], # a HookProvider, traces start here
)
Because Strands already emits the tool name and input on every tool call, the recorder reads them straight off the event, one step per call:
def _on_tool(self, event: AfterToolCallEvent) -> None:
name = event.tool_use["name"] # from Strands' AfterToolCallEvent
self._tool_calls.append({
"tool": name,
"input": event.tool_use.get("input") or {},
"source": TOOL_SOURCE.get(name), # which external source this tool reads
})
Your tools don't change. Strands surfaces what happened through the events; the recorder just assembles it. The pattern works in any agent framework that emits lifecycle events with tool-call data. Strands gives you those events out of the box.
Why Neo4j's own SDK, instead of hand-rolling the graph?
The graph track does not invent a schema. It uses neo4j-agent-memory (Neo4j Labs), the vendor's official reasoning-memory SDK, so the node labels, the writes, and the audit traversal are Neo4j's design, not mine. The recorder for the graph track, Neo4jDecisionRecorder, is the same Strands HookProvider pattern, it just writes each trace into Neo4j through the SDK as the agent runs:
async with memory_client() as client:
trace = await client.reasoning.start_trace(session_id="travel", task=question)
for call in tool_calls:
step = await client.reasoning.add_step(trace.id, thought=..., action=...)
# tag the external source this tool touched, so the audit can traverse to it
await client.reasoning.record_tool_call(
step.id, call["tool"], call["input"], touched_entities=touched)
await client.reasoning.complete_trace(trace.id, outcome=outcome, success=True)
The SDK creates and manages the schema. I never write a CREATE for it.
What should an AI agent audit trail capture?
One decision trace per agent invocation:
question → reasoning steps → tool calls (with inputs) → the sources each call touched
Plus the thing logs never carry: which external source each step touched. That is what makes the reverse audit possible, and it's exactly what a flat log line does not connect.
How do you record decision traces without rewriting your tools?
With the lifecycle hooks your agent framework already emits, shown above. In the demo, the live agent runs its tools and the recorder captures the real steps (the flight search, the fare-alert check): the actual chain, not a plausible story.
For the flat track the trace lives in agent.state, so a session manager persists it. The demo proves this with a real restart: a fresh agent instance restores the same session, and "why did you recommend that flight?" still replays the recorded steps. Without the recorder, the restarted agent recovers 0 steps and confabulates: the session carried the conversation across the restart, but never the tool-by-tool reasoning.
An honesty note the demo states explicitly: "reasoning memory" is an engineering pattern, not an established category in academic memory taxonomies. What research does support is the value of traceability and provenance in agent memory (MemWeaver, the Engram system).
Why store the reasoning at all? Tokens saved, errors avoided
Two payoffs, both measurable. Answering "why did you decide X?" has two paths:
- Read the stored trace. No model call, so zero tokens, and the answer is the real recorded chain, deterministic.
- Ask the model to reconstruct it with no trace. It costs tokens and the answer is confabulated, because the real chain was never kept.
In the demo, replaying from the trace costs 0 model tokens and returns the real steps; asking the model to reconstruct the same chain costs roughly a hundred tokens and invents a plausible story.
So storing the trace saves tokens (no model round-trip to explain a past decision) and avoids errors (the real chain instead of a guess). That is the everyday reason the recorder earns its keep, before you even get to the audit.
The reverse audit: where flat storage breaks
Here's the scenario that separates an audit trail from a pile of logs. The demo runs a live travel-planning session of ten decisions. Some read a fare-alerts feed (picking flights, checking a fare alert); some read only a weather API (best time to visit, what to pack). No hardcoded outcomes, the agent decides on each prompt, and the recorder tags which source each tool call touched.
Then the fare-alerts feed is declared compromised. Which decisions do you need to revisit?
| Store | Reverse audit | Why |
|---|---|---|
| Flat (key-value blobs) | scan every record, one at a time | a flat store has no edges; you read each blob and match on the source it names, and a dependency that ran through another decision's output isn't in the blob at all |
Graph (Neo4j :TOUCHED traversal) |
one query | the SDK records a (:ReasoningStep)-[:TOUCHED]->(:Entity) edge per source, so the audit is a single traversal |
Of the ten live decisions, the graph traversal returns the ones that touched fare_alerts_feed (the flight picks that checked a fare alert, plus the standalone fare-alert checks) and correctly excludes the weather-only decisions. The exact count depends on what the live agent does each run; the property that holds is that the traversal returns every decision whose recorded steps touched the source, and nothing else, at read time.
A quick note on what we are actually measuring
If you have followed the earlier posts, you have seen memory scored on four dimensions (Future AGI, 2026): recall, freshness, contradiction handling, and forgetting. Reasoning memory is not on that list, and it would be dishonest to pretend it is. It does not help the agent recall more or forget better. It is a separate concern: provenance and auditability.
So the metric here is not recall or precision. It is whether, when a source turns out to be wrong, the store lets you find the decisions that touched it.
A flat store can enumerate them too, but only by re-scanning every record on every query, and it cannot follow a dependency that ran through another decision's output. The graph makes that a single traversal it already supports, at any depth. That is a question none of the four standard dimensions ask, which is exactly why it deserves its own demo.
How does the reasoning graph work?
Neo4j's agent-memory SDK creates and manages this schema when the recorder writes a trace:
(:ReasoningTrace)-[:HAS_STEP]->(:ReasoningStep)-[:USES_TOOL]->(:ToolCall)-[:INSTANCE_OF]->(:Tool)
(:ReasoningStep)-[:TOUCHED]->(:Entity)
The entire reverse audit is one query over the :TOUCHED edges:
MATCH (t:ReasoningTrace)-[:HAS_STEP]->(:ReasoningStep)
-[:TOUCHED]->(:Entity {name: "fare_alerts_feed"})
RETURN DISTINCT t.task
You can see the whole graph in Neo4j Browser: point it at the demo's isolated database (:use reasoningdemo), run the demo, and return paths so the Browser draws the edges. The demo ships those Browser queries as trace_graph.VISUALIZE_QUERIES.
Showing the reasoning is not always safe: a privacy note
Being able to replay why the agent decided is useful for audits, but the same trace can leak private data: the tools it called, the inputs it passed (a route, dates, a budget), the sources it read. Treat a decision trace as sensitive:
- Screen what goes into the trace the same way Demo 05 screens what goes into memory. Amazon Comprehend can detect and redact PII in the inputs and evidence before they are recorded.
- Scope the store by tenant / user, so a "why did I decide X?" replay can only read that user's own traces.
- Gate who can replay. "Show me the reasoning" is an audit capability, not a default user affordance; put it behind the same authorization as any other audit log. Neo4j documents access control and auditing for the graph side.
Deterministic vs model-based
The control lives in the agent's harness. The recorder is a Strands HookProvider attached with Agent(hooks=[...]), not a wrapper around the agent.
Recording and auditing are deterministic: assembling the trace from lifecycle events, the SDK's writes, and the :TOUCHED traversal all return the same result for the same recorded input. The one model-based part is upstream: the agent choosing which tools to call as it makes each decision. A model call carries no reproducibility guarantee. Neural-network inference on GPUs varies with floating-point non-associativity and batching, even under greedy decoding (Enabling Determinism in LLM Inference, 2026).
So the set of decisions can differ run to run; the audit over whatever was recorded is exact. An audit trail has to be reproducible even when the thing it audits is not.
Can audit trails be added to an existing agent system later?
Yes, that's the point of the hooks approach. The recorder subscribes to events your agent already emits, so you add hooks=[DecisionTraceRecorder()] (or Neo4jDecisionRecorder()) to the agent constructor and change nothing else. Your tools, prompts, and workflows stay untouched. Traces start accumulating from that moment forward (nothing retroactive).
Two honest scope notes:
- The recorder captures what actually happened (tools called, sources touched, outcome produced). It does not capture the model's internal chain-of-thought, which providers don't expose reliably and which can be unfaithful anyway.
- Start flat, graduate to the graph. If you only ever replay individual decisions, flat state is enough (one lookup). The graph earns its keep when decisions build on other decisions and you need to audit across them.
| Need | Flat store | Neo4j graph |
|---|---|---|
| "Why did you decide X?" (replay) | ✅ one lookup | ✅ one traversal |
| Persistence across restarts | ✅ with a session manager | ✅ database |
| "Source S was wrong, what touched it?" | ⚠️ scan every record, misses indirect dependencies | ✅ one :TOUCHED traversal, at any depth |
| Audit trail for regulated domains | ⚠️ per-decision only | ✅ cross-decision provenance |
The same hooks, the other direction: reusing reasoning to cut cost
This post audits reasoning after the fact. The companion repo stop-paying-for-repeated-llm-calls-sample-for-aws reuses it. Its ReasoningCache is a Strands HookProvider too, but it runs both directions: BeforeInvocationEvent injects a past plan for a similar task (skipping the model round-trip), and AfterInvocationEvent stores the new trajectory. Same events, opposite goal, here we record to ask why later, there they record to avoid re-deciding. If the tokens-saved comparison above interests you, that repo takes it all the way (AWS benchmark: 86% lower cost, 88% lower latency).
Try it
Everything in this post runs from Demo 06 of the companion repo: five tests, the confabulation baseline, the recorder, the live graph recording, the reverse audit, and the tokens-saved comparison. Tests 1, 2, and 5 need only an API key; tests 3-4 also need a graph database. There is a chat_test.py to drive each track from the terminal (--flat / --graph) too.
If your agent's memory (not its decisions) is what needs relationships (multi-hop questions like "who do I know connected to X?"), that's the graph memory post of this series (measured there: vector search 1/4, graph traversal 4/4).
Research referenced
| Paper | Theme |
|---|---|
| MemWeaver | Traceable long-horizon agentic reasoning |
| Less Context, More Accuracy (Engram) | Every stored fact keeps provenance + a supersession chain (preprint) |
We reproduce the mechanism these papers describe (traceability/provenance), not their specific benchmark numbers.
¡Gracias!
Top comments (0)