When an agent gives a wrong answer, the prompt log tells you what it was asked and what it said. It does not tell you which stored facts reached the prompt, which were filtered out, or which were stale. That gap is where agent debugging actually lives.
We build Statewave, a memory runtime for AI agents, so most of what follows comes from instrumenting the retrieval side of that problem. Six layers, ordered from the one everyone already has to the one almost nobody does.
1. Request tracing across the hop boundary
Start here because it is cheap and everything else attaches to it. An agent turn is rarely one process: the framework calls a model, the model requests a tool, the tool calls a memory service, the memory service queries Postgres.
Propagate a request ID across every hop and log it at each. Without it you have four logs and no way to join them, which turns a five-minute investigation into an afternoon.
OpenTelemetry is the standard worth adopting rather than inventing a correlation header. The span boundaries that matter for agents are: turn, model call, tool call, retrieval.
2. Deterministic retrieval, so runs are comparable
This one is a design decision rather than a tool, and it gates the usefulness of everything after it.
If your memory layer searches at query time and samples differently per call, two identical requests return different context. You cannot compare a good run against a bad one because the inputs were never the same. Non-determinism does not just make debugging harder; it makes A/B comparison meaningless.
Our approach is to move the expensive work off the query path: episodes compile into typed memories once per subject change, then context assembly ranks those compiled memories by kind priority, recency, task relevance, temporal validity, and semantic similarity. Same inputs, same bytes.
Whatever memory layer you use, find out whether it guarantees this. If it does not, your first observability investment is making retrieval reproducible, not adding another dashboard.
3. Retrieval receipts
This is the layer that answers "why did the agent say that?"
A receipt is an immutable record of one context assembly: which memories and episodes were selected, the content hash of the assembled bundle, and the policy snapshot in force at the time. Ours are ULID-keyed, which gives chronological sorting at the database level without a separate index.
| { "receipt_id": "01J8XKQ2M7...", "subject_id": "user-42", "selected_entries": ["mem_881", "mem_902", "ep_1204"], "context_hash": "sha256:9f2c...", "canonicalization_version": 3, "policy_snapshot": {"bundle_hash": "sha256:41ab..."} } |
|---|
One detail worth stealing: the canonicalization_version field. Hashing an assembled context is only useful if you can tell later whether a hash mismatch means the content changed or the hashing routine changed. Version the canonicalization and historical hashes stay verifiable.
Prompt logs cannot do this job. A prompt log shows the final string; a receipt shows the decisions that produced it, including which memories were considered and rejected.
4. Replay, with honest semantics
Receipts let you re-run an assembly. What replay means, precisely, matters more than having it.
Ours re-runs against current memories using the original policy bundle captured in the receipt. That is deliberately not byte-for-byte historical reproduction, and the difference is worth stating plainly:
● Memories may have been added, tombstoned, or superseded since. Those appear in the diff as added or removed entries.
● New episodes ingested since will show up in scope.
● Scoring code runs at whatever version is deployed now, so a changed heuristic shows up as a context hash change.
True point-in-time reproduction needs memory snapshots, which we have not built. If you are designing this yourself, decide which semantic you need before building, because retrofitting snapshots is much harder than including them.
5. Health scoring on the subject, not just the service
Service health tells you Postgres is up. It does not tell you that one customer's memory has quietly degraded.
Subject-level health scoring is the agent-specific version of an SLO. Ours computes a deterministic 0 to 100 score from signals already in the data: unresolved sessions, repeated issue patterns, urgency markers, idle open issues, and SLA breaches. Every factor returns its own contribution, so the score is explainable rather than a number nobody trusts.
| # every penalty is named and capped, so a score decomposes _UNRESOLVED_ISSUE_PENALTY = 15 # per open session, capped at 45 _REPEATED_ISSUE_PENALTY = 20 # 2+ sessions sharing a pattern _ESCALATION_PENALTY = 10 # per episode with urgency markers, capped at 20 _IDLE_OPEN_PENALTY = 15 # open issue, no activity in 7+ days _SLA_BREACH_PENALTY = 10 # per breaching session, capped at 20 |
|---|
No ML, no stored state, computed on demand. Determinism is the constraint that keeps it useful: the same data always produces the same score, so a change in the number always means a change in the data.
6. Alert on state transitions, not thresholds
Threshold alerts on a score that moves every request produce noise until someone mutes the channel.
Alert on transitions between named states instead. Ours emits subject.health_degraded when a subject moves healthy to watch, watch to at_risk, or healthy to at_risk, and subject.health_improved on the way back. Unchanged states emit nothing, because the deduplication compares against the last cached state.
That single rule, transitions rather than levels, is the difference between an alert channel people read and one they filter.
Where to start
If you have none of this today, the order that pays fastest:
- Request IDs across hops. One afternoon, and it makes every later layer joinable.
- Make retrieval deterministic. Nothing downstream is trustworthy without it.
- Receipts on retrieval. The single highest-value artifact for debugging agents, and the one most stacks lack.
- Transition-based alerts. Cheap once you have a state model.
Replay and subject health are worth building when you have compliance requirements or a support workflow, and skippable if you do not.
Our implementations of the above are Apache-2.0 and readable in the repo if you want the receipt schema or the health scoring rather than the summary. The provenance and audit trail write-up covers the reasoning behind layer three in more depth.
What does your agent observability stack look like? Particularly curious whether anyone has solved true point-in-time replay without snapshotting the whole store.
Top comments (0)