A flat log can tell you that five things happened. It often cannot tell you which operation caused the next one, which failure triggered a fallback...
For further actions, you may consider blocking this person and/or reporting abuse
There is a fifth shape I would add to the four, because it is invisible in every view except the tree: sibling calls to the same tool with paraphrased inputs and no failure between them. Not a retry (nothing failed), not a fallback (nothing was replaced), just the same question asked again in different words. We found it in our own agent traces around memory retrieval: the agent recalls, gets a plausible answer, and recalls again with a reworded query, sometimes several times, before it acts. Every one of those siblings looks healthy on its own. The cost only shows when you notice that each extra call is also an extra model round, and each round resends the full context.
The check we derived from it is deterministic in your sense: same tool, siblings under one parent, input similarity above a threshold, zero failures in between. That pattern is a policy problem, not a dependency problem, and it responds to a one-line instruction change ("recall once, then act") rather than to any change in the tool. The tree is what made it a countable thing instead of a feeling that the agent was "chatty".
@heinrichneb, that is a useful fifth shape: redundant sibling work without an error edge. I’d make the check rely first on a stable operation or resource key plus normalized arguments, with semantic similarity as an optional signal; otherwise a paraphrase threshold can make a supposedly deterministic rule drift. Recording an explicit purpose or dedupe key at the adapter boundary could make “recall once, then act” enforceable as well as visible.
The stable-key-first ordering is right, and I built it last night to check - it came back empty, in a way that sharpens your caveat rather than contradicting it.
The detector pulled typed key/value pairs out of 735 stored entries: env assignments, pinned versions, host:port, addresses. 58 keys, 56 in agreement, 2 collisions. Both collisions were false. port: looked like a key and is not one - a machine legitimately runs many services on many ports. And the one env-variable clash turned out to be the same variable name in two different systems.
So the key has to be single-valued by nature and carry a scope. OLLAMA_KEEP_ALIVE is not a key; kanzlei:OLLAMA_KEEP_ALIVE and node-1:OLLAMA_KEEP_ALIVE are two. Which is your normalized-arguments point, one level earlier than I had it.
On "recall once, then act" being enforceable: we have an accidental measurement. That line sits in our benchmark's instruction appendix. A config change dropped it without anyone noticing, and recalls per session went from 2.87 to 4.00 on identical tasks. Same corpus, same model. The rule was doing real work, and its absence was invisible until we diffed two runs file by file - which is your argument for recording the intent at the boundary, made by its absence.
@heinrichneb, that measurement is unusually useful because it separates two failure modes: a key without scope creates false collisions, while a missing intent constraint creates silent behavioral drift. I’d encode the first as a composite identity such as
(system, resource, key)and record the second as versioned policy evidence on the trace root. The 2.87 → 4.00 recall shift then becomes a deterministic regression: same task fixture, same model, different policy version, and measurable extra sibling work.Repeated siblings expose only the retries that live above the boundary you wrapped. A
step.toolaround a client that retries internally records one node whether the upstream saw one request or three, so an empty retry shape is not evidence that no retry happened — the same "success hides the cost" the tree was built to surface, one layer down.The deterministic check inherits it: a max-calls rule counts instrumented calls, so the gate stays green while the dependency took three. Worth stating next to the projection list, because the tree is the artifact people will trust for "was a fallback or retry used?".
@vinhnguyenthanhdn, exactly—this is an observability-boundary problem, and “no retry observed” must not be reported as “no retry happened.” If the SDK retries below a wrapped
step.tool, agent-inspect only has evidence for the outer operation unless the adapter or client instrumentation emits each attempt. The clean contract is to expose adapter capabilities and mark retry evidence unavailable when that boundary is opaque, then use explicit attempt metadata when the integration can see it. I’ll make this caveat more explicit because a green max-calls gate without that context would be misleading.The causality gap is real and it's worse than it looks when you've got concurrent tool calls. I'd been chasing a retry bug for two days and the timeline looked fine; it was only when I drew the parent-child relationship on paper that I saw a planning step that had silently spawned a second branch no one was accounting for. The tree structure you're describing is exactly that made automatic, and it's the right unit to debug against. One thing I'm curious about: how does AgentInspect handle steps that share a name across multiple runs, like a polling loop that calls the same tool twenty times?
@hannune, each run is scoped by its own run ID, and repeated steps keep distinct span IDs and parentage even when the display name is identical. A polling loop calling the same tool twenty times should therefore appear as twenty occurrences rather than one collapsed node. A max-calls rule can count those occurrences within the relevant parent. An explicit attempt or iteration attribute is also useful when repetition is intentional; otherwise the tree shows repetition but not the loop’s intended boundary. Thanks for raising that distinction.
Actually 1 of the reasons why I built a custom VC for my IDE and velocity mcp, with a merkle root audit trail. Between the 2 of them, the sitemap + the tool call history creates a clear picture of what changed where, by who and why they did it. Clean intent, clean action, clean credentials, so everything is auditable. Eg. changing a bool to a nullable, would sit in the sitemap's merkle root along with agent context, to say it was made nullable, because of a schema change, along with the tool calls executed to do so. That in turn traces to the schema change, which reveals it was made nullable for a new feature added. That distinction allows you to trace the versions of the app, so if a flaw surfaces, you know when it became a problem and when it was resolved, so you have a window of clients you know you have to patch
@unitbuilds, the Merkle-root provenance is a strong complement to execution tracing. The trace answers what actually ran and where it failed; the versioned root and intent record explain why that code or schema existed. Linking an evidence bundle to the commit or Merkle root would make it possible to move from a runtime failure back to the exact change and rationale that introduced it.
That was the idea, so if anything breaks and gets patched, it's permanent, the history shows exactly why it changes, so if any agent wants to revert the change, they know it wont work and wont even try it, unless they can prove it would work properly.
@unitbuilds, exactly—the stored rationale becomes a guardrail against cycling back into a known-bad state. I would still make the barrier evidence-based rather than absolute: an agent may propose a revert, but it must address the original failing contract and produce new passing evidence before the change is accepted. That keeps history informative without turning an old decision into permanent dogma.
The execution tree framing makes a lot of sense because causality is usually the first thing you lose when agents start delegating, retrying, and running tools in parallel. One thing I’d add is correlating the tree with cost and latency at each node. A fallback might produce a correct answer, but if it consistently adds 3x the tokens or latency, that’s an architectural signal worth surfacing. We’ve run into this kind of issue at IT Path Solutions when looking at production agent workflows the trace becomes much more useful when it explains not just what path the agent took, but what that path actually cost.
@mateo_ruiz_6992b1fce47843, exactly—the tree becomes more useful when duration, tokens, and estimated cost stay attached to each node. I would keep those as facts on the underlying event stream and let the tree project them, so a recovered run can still show that the fallback tripled cost or latency. Aggregate fallback frequency is the production signal I’d watch alongside the single-run view.
The adapter-boundary caveat is the part I would make machine-readable. A trace node should record whether retries are instrumented at the wrapper, client, or transport layer, plus an attempts_observed versus attempts_unknown field. Then a max-calls check can fail closed, or at least surface “incomplete evidence,” instead of treating one successful wrapper span as proof that the dependency was called once. That keeps the execution tree useful without overstating what it can see.
@zira125, yes—visibility coverage needs to be part of the evidence contract. I like a tri-state check result here: pass, fail, or inconclusive.
attempts_observed: 1withattempts_unknown: truemust not satisfymaxAttempts: 1; it should identify the missing instrumentation layer. That lets the tree stay honest about both what happened and what the adapter could actually observe.Flat logs make me invent causality that was never there. When a coding agent retries a tool, spawns a nested plan, and interleaves three "search" calls, the timeline reads as order while the bug was which parent owned the failure. Trees fix the question I actually ask when a run goes sideways: was this a fallback, a parallel sibling, or a second attempt of the same step. Manual step boundaries are annoying to add. They beat scrolling a successful looking log that hid the branch that mattered.
@eduzsh, “the timeline reads as order” captures the failure mode precisely. I’d let adapters infer framework-native boundaries and reserve manual steps for domain transitions the framework cannot know about—approval granted, retrieval accepted, or a durable commit completed. That keeps instrumentation manageable while preserving the parentage needed to distinguish retries, fallbacks, and parallel siblings.
The “path is part of the behavior” line is the whole argument — flat logs treat siblings and children the same, so you debug symptoms instead of the plan that spawned them.
I’ve started forcing a tiny pre-flight before any agent turn: outcome, out of scope, done looks like, never invent. The third line is where the tree should attach — if “done” doesn’t name which child steps must succeed, a fallback can look like progress.
Curious whether your fixtures make it obvious when three tool calls are children of one planning step vs three unrelated roots, or if that still takes a human stare at the tree.
What I find especially interesting here is the distinction between successful output and healthy execution.
We tend to evaluate intelligent systems from the endpoint because the endpoint is visible. Did the agent answer correctly? Did the task complete? Did the workflow return what we expected?
But a successful result can conceal an extraordinary amount of dysfunction upstream.
An agent can fail repeatedly, invoke an unnecessary fallback, duplicate work, consume far more resources than expected, and still hand the user a perfectly acceptable final answer. If we evaluate only the endpoint, all of that disappears into the word success.
The execution tree therefore seems valuable for something larger than debugging: it preserves the difference between what happened and what ultimately worked.
That distinction matters because resilient systems are often designed to absorb failure. But the better a system becomes at recovering from failure, the easier it may become to overlook the failures it is continuously recovering from.
There is almost a paradox there:
Successful recovery can reduce the visibility of the condition that made recovery necessary.
And once failure becomes invisible, “the system still works” can become a dangerously reassuring metric.
I particularly liked your example of retries. Three attempts followed by success and one successful attempt may produce the same visible outcome, but they are not equivalent executions. One tells us only that the task succeeded. The other tells us something about the condition of the system that produced that success.
This also makes your separation between the structured trace and its different projections important. A tree answers one question; checks, diffs, reports, and semantic evaluation answer others. No single representation should be mistaken for the evidence itself.
Perhaps the broader principle is this: when systems become sufficiently good at recovering, observability has to preserve the failures that success would otherwise erase.
Excellent piece, Raju. “Debug the path, not only the answer” is a useful engineering principle — but I suspect it is also a much broader way of thinking about how we evaluate complex systems.
@khalisollis, that recovery paradox is exactly why I think outcome status and execution health have to remain separate. A run can be functionally successful while operationally degraded, so retries, fallbacks, and recovered errors should consume an explicit recovery budget rather than disappear behind the final answer. The trend in that recovery tax is often the earliest signal that a system is normalizing failure.
The execution-tree framing matches what we see measuring agent-SaaS integrations: flat logs can't tell you whether a retry loop was a fallback or a bug. One thing I'd add from our data, the first causal failure is often upstream of the agent entirely (auth/scope issues on the SaaS side). Does agent-inspect distinguish tool-side vs agent-side causes?
@michielinksee, only when the capture boundary provides enough evidence. A tool span can preserve the upstream error, safe metadata, and its parentage, so an adapter may classify an auth or scope failure as tool-side. But agent-inspect should not infer root cause from a generic exception; when provenance is not observable, it should remain unknown rather than being labeled agent-side. Your SaaS example is a good case for a normalized failure-domain field emitted by the adapter. Which distinctions have been most useful in your data—auth, rate limiting, schema errors, or availability?
After a run of ours died between writing its intent row and doing the write, the re-run's row read as a second request, and a weekly cap that counts rows would have charged two slots for one change. We ended up on the stable operation key you suggested in the heinrichneb thread, except ours has to be written before the call rather than derived from it, because our crash landed between the row and the write. What does a killed run leave in the tree for the next one to find?
@rulestack, I’d make the killed run leave an unfinished attempt, not a completed request: the intent node keeps the durable operation key, while the write step has a start event but no terminal event and is reconciled to
interruptedoroutcome_unknown. The next run should link a new attempt to that same logical operation, query the external system or receipt before retrying, and let the cap count one committed operation rather than two intent rows. Did your write target expose any receipt or idempotency lookup, or was your own intent row the only durable evidence?The execution tree makes a lot more sense than trying to reconstruct everything from timestamps once an agent starts doing parallel work.
I also like the distinction between “the run succeeded” and “the path to success was healthy.” A fallback can hide a real problem if you only look at the final result. Having retries and failed branches visible makes it much easier to spot patterns that would otherwise disappear in the logs.
The cost and latency angle is interesting too. It would be useful to see those attached to each node, especially for agents that recover successfully but take several expensive steps to get there.
@pdfpod, that “successful but unhealthy” distinction is the key operational idea. I think a run needs separate functional and path-health outcomes: the answer may pass while retries, fallback count, or node budgets still raise a warning. Per-node duration and token facts make that degradation attributable instead of leaving only an expensive total.
The separation between the underlying structured trace and the tree projection is the key design choice here. When debugging nested agents, flat logs force you to mentally reconstruct scope boundaries across async tool calls, especially when a child agent catches an error and recovers internally without bubbling it up to the orchestrator. Treating the tree as a projection over an immutable event stream preserves the raw timestamps for timing analysis while giving you clean causal boundaries when diffing two runs.
@reidmarlow, agreed—the immutable event stream is what keeps the tree honest. Parent IDs, sequence, timestamps, and terminal state should remain raw facts; trees, timelines, and run diffs are projections optimized for different questions. That also means improving the visual model never requires rewriting the evidence.
That "those two spaces are not decoration" line hit different. Once you see sibling/child ownership in the tree, flat logs feel like trying to read a story where all the sentences are shuffled.
@richard_smith_154156d471ef, that is a good description of the cognitive cost. A timestamp orders events but does not explain ownership, so interleaved siblings look like one narrative even when they belong to different branches. Once parent IDs are explicit, the same evidence can be projected as a tree for causality and a timeline for latency without asking either view to invent the missing relationship.
I like the distinction between a successful run and a healthy run. When building agent tools, I’ve found the hidden retries and fallback paths often matter more than the final status.
@julianneagu, exactly—the final status compresses away recovery debt. I’d keep
task_outcomeseparate frompath_health, then let retries, fallback use, and budget overruns raise warnings or fail contracts even when the answer passes. That makes a recovered run comparable to a clean one instead of treating both as identical success.Agree completely — flat logs lose the causal structure that actually matters when an agent fans out into parallel tool calls. We hit the same wall and ended up modeling agent runs as a tree of typed events rather than log lines, which made "which branch failed and why" answerable by inspection. The "execution tree" name nails it.
@solonjava, typed events are what keep the tree from becoming another attractive but ambiguous log view. Parent IDs provide ownership; event kind, status, and failure domain let the renderer distinguish parallel work, retries, and recovered errors without guessing from names. I also prefer preserving the event stream immutably so the same evidence can become a causal tree or a latency timeline. How do you maintain parentage when a child branch crosses a process or service boundary?
Excellent point on why flat logs fail for agent causality. Using execution trees to explicitly map nested work, fallbacks, and repeated sibling operations gives a much clearer picture of the agent's path. It is interesting how this visual structure makes it easier to convert suspicious patterns into deterministic checks, like validating maximum tool call limits. Great approach to debugging the actual behavior rather than just the final output.
@mindinu, turning visible shapes into checks is where the tree becomes more than a visualization. A maximum-call rule is useful, but I’d scope it to a logical parent and tool identity so legitimate parallel calls elsewhere do not trip it. The same event stream can then support run diffs that show when retry or fallback structure changes between versions.
The sibling repeat pattern for retries is the one I kept wishing for when debugging agents last year. You see a green success, then spend an hour figuring out it only got there on the third attempt after two silent upstream timeouts. Worth adding to the check layer: a retry budget assertion that fails the run if attempts-observed exceeds the expected ceiling, so you don't accidentally interpret "recovered successfully" as "recovered cleanly."
@hannune, yes—a retry budget belongs in the structural checks. I’d model both a per-step
maxAttemptsand a run-levelmaxRecoveredFailures, with explicit allowances for expected fallbacks. Then a green answer can still fail or warn on path health before silent upstream timeouts become the accepted baseline.