DEV Community

Shashank Agarwal
Shashank Agarwal

Posted on Edited on Fully Autonomous

How to Investigate Agent Failures With Captured Traces

Updated October 29, 2026. Trace analysis now refers to captured operations and testable explanations, rather than guaranteed access to private reasoning or automatic proof of a root cause.

A trace is most useful when it lets a reviewer move from an observed failure to the operations that could explain it. The goal is to identify a testable change, not to infer certainty from a plausible story.

Quick answer: Define the intended outcome, inspect the available operations in order, compare tool inputs and outputs with the task, identify competing explanations and test a bounded correction. Record gaps in the evidence. A missing operation cannot be reliably reconstructed from an aggregate score.

Follow the recorded operations

OpenTelemetry describes traces as a way to follow a request through recorded operations represented by spans. What your application exposes depends on instrumentation, integration support and configuration.

Start by identifying the user’s task and the application version. Then inspect the relevant model, retrieval and tool boundaries that were captured. Do not assume that every private thought, every uninstrumented dependency or every downstream effect is present.

Use a concrete investigation record

Suppose an illustrative refund workflow proposes the wrong amount. The captured tool result includes a listed price of 99.99 and a discount of 10.00. That alone is not enough to determine the refund: the applicable policy, taxes, shipping and previous adjustments may matter.

Before changing a prompt, retain the actual policy and the amount the application computed or proposed. Separate these questions:

  1. Did the correct order reach the tool?
  2. Did the result contain the required fields?
  3. Was the applicable refund rule available?
  4. Did an explicit calculation apply that rule correctly?
  5. Did the proposed action match the approved result?

The values above are teaching inputs, not customer data or a universal refund rule.

Keep alternative explanations alive

A wrong amount could reflect incomplete source data, the wrong policy, an arithmetic error or a mismatch between the calculated value and the action. The final answer alone does not distinguish them.

Write each candidate explanation next to the evidence that supports or weakens it. If evidence is missing, improve the relevant instrumentation or reproduce the behavior in a controlled test. Avoid adding broad payload capture when a smaller field or identifier answers the investigation question.

Test the proposed correction

If the application needs a deterministic calculation, move that calculation into an explicit, testable operation. Exercise cases with and without discounts, missing policy data and relevant prior adjustments. Confirm that the agent handles unavailable information without inventing an amount.

Keep the application version, evaluation setup and observed results together. A recommendation should lead to a candidate test; release remains an engineering decision.

I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. Noveum connects captured operations, prepared evaluation items and investigation evidence. See the workflow walkthrough and production monitoring guide.

Follow my work on LinkedIn. This refresh was drafted with AI assistance.

Top comments (0)